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Executive  Summary 


This  report  summarizes  the  first  part  of  a  two-part  study  performed 
for  the  Division  of  Health  Evaluation,  Office  of  the  Assistant  Secretary 
for  Planning  and  Evaluation,  Department  of  Health,  Education,  and  Welfare 
by  the  Technical  Analysis  Division  (TAD)  of  the  National  Bureau  of 
Standards.    The  study  reviewed  the  first  two  years  (July  1969  to  October 
1971)  of  a  proficiency  testing  program  for  laboratories  licensed  under 
the  Clinical  Laboratory  Improvement  Act  (CLIA)  of  1967  on  the  basis  of 
data  collected  under  the  regulatory  authority  of  the  Center  for  Disease  . 
Control  (CDC). 

The  major  study  effort  was  associated  with  statistical  analyses  of 
the  data  to: 

(1)  Evaluate  the  temporal  effect  of  continued  participation  in  the 
CDC  proficiency  testing  program. 

(2)  Describe  the  population  of  licensed  (interstate)  laboratories 
in  terms  of  various  profile  descriptors. 

(3)  Determine,  through  one-way  analyses  of  variance,  whether  the 
mean  differences  associated  with  the  various  levels  of  each 
independent  variable  (excluding  number  of  employees)  were 
significantly  different. 

(4)  Contrast  the  relative  performance  of  licensed  and  volunteer 
laboratories . 

(5)  Detemine  if  bias  is  introduced  by  pooling  various  analytical 
methods  in  calculating  the  reference  laboratory  sample  target 
value. 

Statistical  analyses  were  performed  on  two  sets  of  data  supplied  by 
CDC,  the  agency  within  the  Public  Health  Service  (PHS)  responsible  for 
administering  the  CLIA  '67  program.    The  first  set,  covering  268  licensed 
clinical  laboratories,  contained  the  analytical  results  (raw  data,  not 
scores)  of  proficiency  tests  conducted  in  clinical  chemistry  and  hematol- 
ogy.   Also  contained  in  this  set  were  the  results  of  laboratory  testing 
for  several  hundred  other  clinical  laboratories  who  participate  in  the 
same  proficiency  tests  but  who  are  not  licensed  under  the  CLIA  program 
(volunteer  and  reference  laboratories). 

The  second  set  of  data  contained  profile  information  on  237  of  the 
268  licensed  laboratories,  including  a  record  of  the  number  of  full-time 
employees,  the  highest  academic  achievement  level  of  the  working  super- 
visors, and  the  total  number  of  accreditation  or  licensure  programs  in 
which  the  laboratory  was  involved.    Corresponding  profile  data  were  not 
available  for  the  volunteer  or  reference  laboratories. 

The  temporal  effect  was  evaluated  through  pooling  the  data  over  real- 
tijne.     In  this  manner,  laboratories  were  grouped  according  to  their 
absolute  point  on  the  learning  or  experience  curve.    Inference  to  the 
performance  of  individual  laboratories  was  not  considered,  nor  was  it 
ever  intended  to  be  a  part  of  this  study.    Also  inherent  in  the  analyses 
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was  the  assumption  of  methods  consistency  independent  of  the  level  of 
each  constituent.    The  major  findings  of  the  analysis  are  as  follows: 

1.  It  was  not  possible  to  discern  meaningful  changes  over  the  two- 
year  period  in  the  overall  level  of  licensed  or  volunteer 
laboratory  performance  as  measured  by  the  mean- accuracy  approach 
(average  relative  difference  from  target  value  established  by 
CDC  reference  laboratories)  for  any  of  the  13  constituents 
analyzed. 

2.  The  variability  among  the  licensed  laboratories  as  measured  by 
the  standard  deviation  has  decreased  over  the  two-year  period 
under  study.    This  is  an  indication  that  the  group  performance 
has  become  more  consistent.    The  same  remark  is  true  for  the 
volunteer  laboratory  group  as  well,  where  the  improvement  was 
often  much  more  marked. 

3.  Of  the  four  independent  variables  studied,  including  Accredita- 
tion, State  Program  Intensity,  Supervisory  Education,  and 
Analytical  Method/Laboratories,  only  the  last  one  appeared  to  be 
significant.    That  is,  the  general  level  of  capability  was  about 
the  same  whether  laboratories  were  or  were  not  involved  in  pro- 
ficiency testing  programs  other  than  CDC;  whether  they  were 
located  in  states  which  did  or  did  not  have  proficiency  testing 
programs  of  their  own;  or  whether  the  working  supervisor  had  a 
B.S. ,  M.S.,  Ph.D.,  or  M.D.  degree.    Only  the  Analytical  Method 
used  (or  the  difference  among  laboratories)  appeared  to  be  signif- 
icant (p  <  .001). 

4.  The  data  do  not  appear  to  support  an  argument  in  favor  of 
establishing  reference  group  target  values  on  a  method- by-method 
basis.    More  data,  collected  through  the  use  of  a  statistically 
designed  experiment  which  would  avoid  confounding  analytical 
methods  with  laboratories  by  having  each  laboratory  use  several 
methods,  would  be  helpful  in  reaching  more  rigorous  and  definitive 
conclusions . 

The  Clinical  Laboratory  Improvement  Act  of  1967,  administered  by 
the  Center  for  Disease  Control,  provides  for  a  many- faceted  interlocking 
regulatory  program.    This  study  has  looked  at  only  one  aspect  of  the 
CLIA  '67  program,  namely  proficiency  testing.    Not  included  in  the  study, 
but  also  of  major  importance,  are  such  functions  as  on-site  laboratory 
reviews,  training,  written  critiques  in  the  qualitative  specialties,  and 
method  development.    There  are  few  indications  of  meaningful  changes  in 
the  average  level  of  performance  as  measured  by  proficiency  testing 
results  since  the  licensed  laboratories  entered  the  CLIA  proficiency 
testing  program.    Consideration  should  be  given  to  alternative  sampling 
methods,  such  as  reduced  or  skip-lot  sampling,  for  those  constituents 
which  appear  to  present  no  analytical  challenge  to  the  licensed  and 
volunteer  laboratories.    Greater  emphasis  might  then  be  placed  on  those 
constituents  which  give  the  laboratories  the  most  difficulty  (cholesterol 
and  creatinine,  for  example). 
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This  report  includes  a  review  of  the  legislative  intent  behind  the 
law  (CLIA  '67)  and  an  annotated  bibliography  of  pertinent  literature 
in  the  field  of  laboratory  proficiency  testing  for  the  years  1963  to  1971. 
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Clinical  Laboratory  Performance  Analysis 
Using  Proficiency  Test  Statistics 

Peter  W.  Finkel 
John  W.  Rowen 

The  proficiency  testing  aspects  of  the  Clinical  Laboratory 
Improvement  Act  of  1967  program  was  assessed.    The  overall 
ability  of  licensed  or  volunteer  laboratories  to  accurately 
determine  mean  values  for  any  of  the  13  constituents  was  not 
significantly  altered  during  the  first  two  years  of  program 
operation.    However,  the  variability  of  the  laboratories  has 
decreased  over  the  two-year  period.     It  appears  that  the 
program  has  increased  the  consistency  of  laboratory  performance. 

The  general  level  of  laboratory  capability  seemed  to  be 
independent  of  involvement  in  state -supported  or  voluntary 
proficiency  testing  programs  other  than  the  CDC  program  and 
of  whether  the  working  supervisor  had  a  B.S. ,  M.S.,  Ph.D.,  or 
M.D.  degree.    Choice  of  analytical  method  did  significantly 
affect  performance.    Although  insufficient  evidence  was 
available  to  make  a  definitive  statement,  the  data  do  not 
appear  to  support  arguments  favoring  establishment  of  method- 
dependent  reference  group  target  values. 

Finally,  it  appears  that  consideration  should  be  given 
to  alternative  sampling  methods,  such  as  reduced  or  skip-lot 
sampling,  for  those  constituents  which  appear  to  present  no 
analytical  challenge  to  the  licensed  and  volunteer  laboratories. 
Greater  emphasis  might  then  be  placed  on  those  constituents 
which  give  the  laboratories  the  most  difficulty  (cholesterol 
and  creatinine,  for  example). 

Key  words:  Accuracy;  CLIA  '67;  clinical  chemistry;  hematology; 
microbiology;  precision;  proficiency  testing. 


1 .  Introduction 

This  report  summarizes  the  first  part  of  a  two-part  study  of  clini- 
cal laboratory  performance  in  the  United  States.    The  study  was  conducted 
by  the  Technical  Analysis  Division  (TAD)  of  the  National  Bureau  of 
Standards  for  the  Division  of  Health  Evaluation,  Office  of  the  Assistant 
Secretary  for  Planning  and  Evaluation,  Department  of  Health,  Education, 
and  Welfare. 

The  objectives  of  this  study  were  to  determine  if  the  participation 
of  interstate  (and  volunteer)  laboratories  in  the  CDC  proficiency  testing 
program: 

(1)    Has  resulted  in  significant  changes  in  average  laboratory 

performance  as  measured  by  the  accuracy  (distance  from  target 
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value)  of  proficiency  testing  results  for  these  laboratories 
over  a  two-year  period. 
(2)    Reveals  classes  of  laboratories  which  are  more/less  accurate 
from  the  standpoint  of  proficiency  test  results. 
,        (3)    Offers  insights,  conclusions  or  recommendations  pertaining  to 
the  frequency  of  testing,  constituents  tested  and  laboratory 
analysis  procedures  used. 

The  purpose  of  this  report  is  to  publish  the  results  of  statistical 
analyses  performed  on  an  extensive  data  base  \\^ich  was  compiled  and 
maintained  by  the  Center  for  Disease  Control  (CDC) ,  Public  Health 
Service,  Health  Services  and  Mental  Health  Administration,  Department  of 
Health,  Education, and  Welfare.    The  data  for  these  analyses  were  the 
results  of  proficiency  tests  administered  by  the  Center  for  Disease 
Control,  extending  for  a  two-year  period  beginning  in  July  1969;  and 
various  profile  descriptors  of  the  laboratories  licensed  under  CLIA  '67. 


2.  Background 

Public  Law  90-174  (Partnership  for  Health  Amendments,  1967), 
Section  5,  was  enacted  to  assure  consistent  performance  of  accurate 
laboratory  procedures  and  services.    The  law  also  includes  provisions 
relating  to  the  application  for  and  issuance  of  licenses,  quality  control, 
personnel  standards  setting,  and  proficiency  testing.    The  legislation 
impacts  on  clinical  laboratories  engaged  in  interstate  commerce.  The 
law  was  enacted  in  1967,  but  was  not  fully  implemented  until  July  1,  1969. 
The  Center  for  Disease  Control  serves  as  the  regulatory  agency  for  this 
legislation. 

Clinical  laboratories  that  meet  eligibility  requirements  under  the 
Clinical  Laboratory  Improvement  Act  (CLIA)  legislation  are  licensed  by 
"procedure  or  category  of  procedure"  such  as:    microbiology,  serology, 
bacteriology,  glucose,  hemoglobin,  etc.    After  receiving  their  license 
they  are  thereafter  subject  to  periodic  proficiency  testing  in  many  of 
the  licensed  areas.    These  tests  are  used  as  one  means  for  determining 
the  competency  of  the  laboratory  staff  and  the  adequacy  and  quality  of 
facilities,  equipment,  reagents  and  laboratory  procedures.  Normally, 
samples  for  the  proficiency  testing  program  are  prepared  and  mailed  to 
the  licensed  laboratory  every  third  month.    The  analytical  results 
obtained  by  the  laboratory  are  subsequently  returned  to  CDC  for  scoring, 
evaluation,  and  critique  where  applicable.     (Appendix  D  provides  a  brief 
description  of  the  program.)    When  a  laboratory's  performance  reflects 
lack  of  proficiency  (significant  deviation  from  norms  established  by  the 
CDC  reference  laboratories  and/or  peer  group  analysis)  corrective  measures 
are  taken  by  CDC.    The  extent  of  the  corrective  action  varies  according 
to  the  individual  situation.    In  instances  where  poor  performance  occurs 
repeatedly,  where  cooperation  is  lacking  or  suggested  remedial  action  is 
not  taken  by  the  laboratory,  suspension  or  revocation  of  the  license  can 
result. 
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Appendix  A  of  this  report  provides  a  brief  summary  of  events  leading 
to  the  enactment  of  CLIA  '67,  changes  in  the  administration  of  the  act 
which  have  occurred  since  its  enactment,  and  details  concerning  the  pur- 
pose and  sub- objectives  of  the  program  as  administered  by  CDC. 


2.1.    Proficiency  Testing  Programs 

Proficiency  testing  programs  are  monitoring  systems  designed  to 
provide  a  basis  for  ascertaining  the  reliability  of  laboratory  performance. 
These  programs  serve  to  prompt  the  appropriate  actions  needed  to  assure 
results  that  are  medically  useful  and  interconvertible  (the  results  from 
one  method  can  be  translated  to  comparable  results  from  a  different 
method) .    Proficiency  testing  often  involves  the  analysis  of  submitted 
samples;  however,  it  is  important  to  remember  that  this  is  but  one  com- 
ponent of  any  total  proficiency  assurance  program. 

Some  other  objectives,  or  components,  of  a  multifaceted  program  such 
as  CLIA  '67  are: 

a.  To  define  the  "state-of-the-art"  in  terms  of: 

(1)  Methodology 

(2)  Reagents 

(3)  Standards 

(4)  Personnel  performing  test 

b.  To  define  sources  of  error  in  terms  of: 

(1)  Procedure 

(2)  Education  and  performance  of  analyst 

(3)  Reagent 

(4)  Equipment  (includes  improper  use  of  equipment  as  well  as 
use  of  improper  equipment  for  an  assay) 

(5)  Monitoring  systems 

c.  To  define  criteria  to  substantiate  "medical  usefulness" 

d.  To  define  educational  needs  and  stimulate  remedial  measures. 

The  Federal  Regulations  published  on  December  31,  1968,  clearly 
describe  CDC's  regulatory  authority  with  regard  to  unacceptable  perform- 
ance by  a  licensed  laboratory  in  the  proficiency  testing  program.  The 
reader  is  referred  to  the  published  regulations  for  a  complete  description 
of  this  topic,  but  in  brief,  it  is  incumbent  upon  a  licensed  laboratory 
to  maintain  satisfactory  performance  in  all  proficiency  testing  programs 
in  which  it  is  involved.    Failure  to  do  so  can  result  in  revocation  of 
licensure  or  exemption  therefrom. 

The  extent  to  which  the  existing  proficiency  programs  do,  in  reality, 
impact  on  general  laboratory  performance  improvement  is  still  not  widely 
agreed  upon.    The  difficulty  in  determining  whether  proficiency  testing 
programs  have  improved  performance  is  due  to  one  or  more  of  the  following: 

a.    Changes  in  methods  of  interpretation  of  the  results. 
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b.  The  effect  of  new  laboratory  entries  in  the  program  whose 
initial  results  obscure  the  improved  changes  of  the  established 
participating  members. 

c.  Technological  changes  in  the  laboratory  testing. 

d.  The  fact  that  virtually  all  proficiency  testing  programs, 
including  CLIA  '67,  require  laboratories  to  analyze  samples  and 
specimens  which  are  identifiable  as  testing  material.  The 

:  .   ■         laboratory  that  knows  it  is  being  tested  can  use  more  experienced 
personnel,  better  quality  control  procedures  and  more  accurate 
testing  procedures  for  these  sample  analyses.    In  some  cases, 
the  licensed  laboratory  has  split  the  sample  received  from  CDC, 
analyzed  one  portion  itself,  and  sent  the  other  portion  to 
another  laboratory  for  confirmative  analysis.    Thus,  proficiency 
testing  can  result  in  measures  which  are  representative  of  the 
best  work  that  a  laboratory  is  capable  of  instead  of  its  routine 
performance. 

A  review  of  the  literature  indicates  that  it  has  been  extremely 
difficult  to  determine  whether  these  programs  have  indeed  accomplished 
the  goal  of  improA/ing  laboratory  performance.    There  is  a  paucity  of 
data  and  much  of  what  is  available  is  conflicting.    For  example,  profi- 
ciency test  data^  from  the  College  of  American  Pathologists  (CAP) 
indicates  a  significant  improvement  over  the  last  20  years.  Proficiency 
testing  data  obtained  by  the  New  York  City  Health  Department^,  however, 
indicates  no  significant  improvement,  albeit  for  a  shorter  period  of  time, 
over  the  years  1964-1968.    Other  published  studies ^  support  the  New  York 
City  findings  by  demonstrating  the  inability  of  a  laboratory  to  perform 
equally  well  in  two  different  proficiency  testing  programs. 

The  aforementioned  findings  and  speculations  should  be  considered 
when  proficiency  testing  is  suggested  as  anything  other  than  one  of 
several  indices  to  be  used  for  laboratory  evaluation.    Perhaps  a  more 
germane  question  to  ask  is  whether  proficiency  testing  in  combination 
with  other  measurement  and  control  procedures  has  any  utility. 


iL.  P.  Skendzel,  D.  J.  Hanson,  and  W.  H.  Civen,  "The  1949  College  of 
American  Pathologists  Survey  Revisited,"  American  Journal  of  Clinical 
Pathology,  lIV  (1970),  493. 

2m.  Schaeffer,  D.  Wedelock,  P.  S.  May,  S.  Blatt,  and  M.  Wilson,  "The 
Clinical  Laboratory  Improvement  Program  in  New  York  City  II,  Progress 
After  Five  Years  of  Experience,"  Health  Laboratory  Science,  VII  (1970), 
242. 

3r.  G.  Martinek,  et  al.,  "Comparison  of  Methods  for  Conducting  Laboratory 
Proficiency  Surveys,"  Health  Laboratory  Science,  V  (1968),  239. 


4 


3.    Study  Methodology 


3.1.    Inputs  to  the  Analysis 

The  analysis  was  performed  on  data  compiled  and  maintained  by  the 
Center  for  Disease  Control.    The  data  were  of  two  types: 

a.  Profile  information  for  most  of  the  licensed  (interstate) 
laboratories,  including  types  of  accreditation  or  licensure 
programs  in  which  active,  state  in  v^ich  located,  number  of 
full-time  employees,  number  and  academic  training  of  working 
supervisors . 

b.  Proficiency  test  results  for  licensed,  voluntary,  and  reference 
laboratories  covering  the  period  of  July  1969  through  September 
1971.    These  data  pertain  to  the  following  constituents  of  human 
blood:    glucose,  creatinine,  urea  nitrogen,  uric  acid,  total 
protein,  cholesterol,  calcium,  sodium,  potassium,  chloride, 
hematocrit,  red  blood  cell  count  (RBC)  and  hemoglobin.  The 
analytical  method  used  by  the  laboratory  is  also  reported.     If  a 
laboratory  is  licensed  in  a  category  (e.g.,  clinical  chemistry) 
or  subcategory  (e.g.,  cholesterol)  for  \\diich  CDC  has  developed  a 
proficiency  testing  program,  it  receives  a  shipment  of  test  samples 
every  three  months.    The  general  procedure  calls  for  three  samples 
per  shipment,  usually  one  at  each  of  the  high,  medium  and  low 
level  of  each  constituent. 

In  addition  to  the  technical  data,  information  was  obtained  on  the 
"strength"  of  legislation  and  controls  within  states  in  which  interstate 
and  volunteer  laboratories  are  located.    An  extensive  literature  search 
(Appendix  B)  was  also  conducted  to  gather  information  for  the  development 
of  a  suitable  analysis  design  and  for  the  interpretation  of  results. 


3.2.  Method 

The  methods  used  for  analysis  of  the  data  were  relatively  simple, 
with  the  major  activities  devoted  to  re- formatting  to  detect  discernible 
temporal  effects,  editing,  and  augmentation  of  basic  data  already  compiled 
on  magnetic  tape.    Computer  programs  were  developed  to  group  and  statis- 
tically analyze  this  information  as  follows: 

a.  The  raw  data  (analytical  test  results)  were  separated  on  a 
const ituent-by- constituent  (test-type)  basis. 

b.  A  target  value  was  calculated  for  each  sample,  using  the  reference 
laboratory  analyses.     In  each  instance  where  there  were  at  least 
30  reference  laboratories  reporting  results  on  a  method- by-method 
basis,  the  target  value  consisted  of  the  average  analysis  for 
those  30  or  more  reference  laboratories.    When  there  were  fewer 
than  30  reference  laboratories  using  one  particular  method,  the 
data  were  pooled  and  one  target  value  was  calculated  for  the 
methods  involved.    That  one  target  value  was  then  used  for  all  of 
the  methods  which  had  been  pooled. 
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c.  The  difference  between  the  analytical  result  reported  by  a 
laboratory  in  the  test  and  the  target  value  calculated  from  the 
reference  laboratory  data  was  used  as  the  measure  of  performance. 
The  difference  was  normalized  by  dividing  it  by  the  target  value. 
For  example,  a  licensed  laboratory  analyzed  a  sample  of  choles- 
terol by  method  A  and  reported  185  mg/100  ml  as  its  result.  The 
average  reference  laboratory  analysis  (calculated  from  63  differ- 
ent laboratories  all  using  method  A)  was  166.5  mg/100  ml.  The 
difference  (185  -  166.5  =  18.5)  was  divided  by  the  target  value; 
18.5/166.5  =  0.11,  and  the  result  was  the  "normalized"  measure  of 
performance  for  that  laboratory  for  that  sample. 

d.  Each  laboratory's  data  file  was  searched  in  order  to  determine 
when  it  first  entered  the  CDC  proficiency  testing  program.  This 
entry  date  was  designated  "time  zero"  for  that  laboratory,  estab- 
lising  its  position  in  a  pseudo-time  scale. 

e.  The  normalized  difference  (performance)  data  for  interstate 
laboratories  were  grouped  according  to  the  pseudo-time  scale,  and 
group  averages  were  calculated.    That  is,  all  of  the  differences 
representing  "first"-  analyses  of  cholesterol  were  pooled  together 
(by  method  of  analysis)  to  give  an  average  difference  for  "time 
zero."    The  same  procedure  was  followed  for  each  subsequent  time 
period.    This  data  is  summarized  in  Table  C-1,  Appendix  C. 

f .  The  volunteer  laboratory  data  were  treated  exactly  as  was  described 
in  the  preceding  paragraph,  with  one  exception.    Whereas  the 
first  entry  into  the  data  bank  for  almost  all  of  the  licensed 
laboratories  actually  represented  their  first  sample  analyses , 
there  are  a  small  number  of  licensed  laboratories  which  were 

r         participating  in  the  CDC  proficiency  testing  program  prior  to  the 
enactment  of  CLIA  '67.    Hence,  their  "first"  analyses  were  at  a 
much  earlier  time,  and  these  records  are  not  part  of  the  CLIA  '67 
proficiency  testing  data  base.    Consequently,  even  though  the 
volunteer  laboratory  data  covers  the  same  time  frame  as  the  inter- 
state data,  it  does  not  represent  the  same  segment  of  the  overall 
.  learning  or  experience  curve.    This  data  is  summarized  in  Table 
C-2,  Appendix  C. 

Target  values  established  by  CDC  reference  laboratories  were  used  as 
the  standard  of  comparison  in  this  study.    No  analysis  has  been  done, 
however,  to  test  the  validity  of  these  target  values.    It  might  be  inform- 
ative to  point  out  some  characteristics  of  the  reference  laboratory  group. 
For  example,  a  laboratory  which  is  licensed  in  one  or  more  categories  may 
serve  as  a  reference  (or  referee  for  the  qualitative  specialties)  labora- 
tory in  areas  in  which  it  is  not  licensed.    Many  of  the  CDC  reference 
laboratories  rank  among  the  leading  state  health  departments  and  teaching 
hospitals.    There  do  not  appear  to  be  any  prescribed  ground  rules  for 
selecting  reference  laboratories  except  that  they  must  be  willing  to 
analyze  the  samples  and  report  the  results  back  to  the  originating  source. 

The  computer  programs  used  in  this  study  provided  the  following 
outputs : 
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1.  A  temporal  analysis  of  laboratory  proficiency  for  all  partici- 
pating laboratories  for  each  constituent  test  method.    If  labora- 
tories were  "improving"  over  time,  mean  differences  from  the 
estimated  target  values  should  have  become  smaller.    If  no 
"improvement"  occurred,  convergence  would  not  have  been  noted. 

2.  An  analysis  of  variance  program  determining  whether  performance 
(accuracy  with  which  laboratories  approximated  target  values)  is 
dependent  upon  laboratory  characteristics  such  as  size,  accredit- 
ation, and/or  academic  training  of  the  supervisor. 

These  outputs  were  analyzed  and  are  presented  in  this  report,  together 
with  conclusions  upon  which  to  base  future  action  on  the  part  of  the 
Department  of  Health,  Education,  and  Welfare. 


4.    Analytical  Results 

4.1.    Temporal  Effect 

Participation  in  a  proficiency  testing  program  may  result  in  changes 
in  individual  laboratory  performance.    Changes  in  overall  group  perform- 
ance are  not  as  easily  discerned.    However,  there  are  at  least  two  ways 
in  which  changes  among  the  laboratories  might  be  detected.    An  analysis 
of  the  general  level  of  accuracy  which  clinical  laboratories  achieve  as 
a  consequence  of  continued  participation  in  a  proficiency  testing  program 
would  provide  one  indication  of  change.    The  "scatter"  (standard  deviation) 
of  results  obtained  among  the  participating  laboratories  would  be  a 
second  indication.    As  the  group  of  laboratories  becomes  more  homogeneous 
the  between- laboratory  standard  deviation  will  be  smaller. 

Several  assumptions  are  inherent  in  the  statistical  interpretation  of 
the  data.    The  following  assumptions  are  representative: 

1.  The  distribution  of  the  relative  differences  between  a  laboratory's 
result  and  the  target  value  is  not  functionally  related  to  the 
analytical  method  used,  the  level  of  the  constituent,  or  the 
constituent  itself,  except  for  a  possible  learning  effect  ■l^^ich 
may  vary  from  one  constituent  to  another. 

2.  The  normalizing  technique  employed,  whereby  the  observed  differ- 
ence between  a  laboratory's  result  and  the  target  value  is 
divided  by  the  target  value  [(0-E)  •^  E] ,  satisfactorily  negates 
any  effect  which  the  level  of  the  constituent  might  have  had  on 
the  difference. 

3.  The  groups  of  laboratories  (the  size  of  i\diich  increased  by  a 
factor  of  at  least  three  during  the  two  years  under  study)  are 
equally  representative  at  each  point  in  time.    That  is,  those 
laboratories  which  entered  the  interstate  licensure  program  early 
in  its  development  do  not  vary  significantly  in  their  general 
level  of  performance  from  laboratories  which  entered  the  program 
at  a  later  date.    This  assumption  also  implies  that  early  or  late 
entrance  into  the  program  is  not  associated  with  the  analytical 
methods  used. 
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Paragraphs  4.1.1  and  4.1.2  discuss  measures  o£  performance  for  both 
the  interstate  and  volunteer  laboratories. 


4.1.1.    Interstate  Laboratories 

This  phase  of  the  statistical  analysis  examined  the  question  of 
whether  participation  in  CDC's  proficiency  testing  program  results  in  a 
changed  level  of  performance  over  time  for  laboratories  in  general.  The 
question  was  examined  on  the  basis  of  overall  accuracy  at  various  time 
points  after  entry;  i.e.,  by  the  mean  performance  among  participating 
laboratories  and  by  the  consistency  among  the  participating  laboratories, 
at  various  time  points  after  entry  into  the  program.    Improved  performance 
would  be  reflected  by  smaller  mean  differences  and  by  smaller  standard 
deviations,  respectively,  with  continued  program  participation.    A  linear 
regression  analysis  was  performed  on  these  values,  for  each  constituent, 
over  the  two-year  period  1969-1971. 

The  constituents  for  which  quantitative  data  was  available  and  for 
which  the  regression  analyses  were  performed  were:    glucose,  creatinine, 
urea  nitrogen,  uric  acid,  total  protein,  cholesterol,  triglycerides, 
calcium,  sodium,  potassium,  chloride,  hematocrit,  red  blood  cell  count, 
and  hemoglobin. 

On  the  basis  of  nine  or  less  quarters  of  testing,  it  was  not  possible 
to  identify  significant  shifts  in  mean  differences  for  any  constituents 
at  the  five  percent  level  of  probability. 

The  analysis  also  identified  a  phenomenon  which  has  not  been  satis - 
factorally  explained.    Sometimes  there  was  a  consistent  (but  not  statis- 
tically significant)  bias  between  the  mean  differences  for  licensed 
laboratories  and  reference  laboratory  target  values.    For  example,  Table 
C-1  (Appendix  C)  shows  that  when  creatinine  was  analyzed  by  the  alkaline 
picrate  (w/o  Lloyds)  method,  the  target  value  (the  average  analysis  for 
all  the  reference  laboratories  which  used  that  same  method)  was  always 
larger  than  the  mean  computed  for  the  interstate  group  using  the  same 
method.    This  was  also  true  for  uric  acid  when  analyzed  by  the  phospho- . 
tungstate  carbonate  method  and  cholesterol  analyzed  by  the  AutoAnalyzer 
and  SMA  methods.    Positive  biases  (i.e.,  the  target  value  was  consistently 
smaller)  were  not  as  prevalent;  they  were  observed  only  in  the  following 
instances:     (1)  urea  nitrogen,  all  methods;  (2)  red  blood  cell  count,  all 
methods;  (3)  hematocrit,  specifically  the  micro  Hematocrit  method,  as  well 
as  for  all  methods;  and  (4)  chloride.    No  definitive  information  has  been 
found  to  explain  these  biases. 

The  variability  of  the  interstate  laboratories,  as  demonstrated  by 
the  pooled  standard  deviations,  indicates  that  the  group  is  becoming  more 
homogeneous.    From  Table  C-1,  Appendix  C,  it  is  evident  that  most  of  the 
standard  deviations  become  smaller  as  time  in  the  proficiency  testing 
program  increases.    Exceptions  to  this  trend  are  found  for  creatinine  and 
red  blood  cell  count. 
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4.1.2.    Volunteer  Laboratories 


Volunteer  laboratories  were,  on  the  average,  performing  at  slightly- 
lower  proficiency  levels  for  some  constituents  in  1971  than  they  were  in 
July  1969,  (i.e.,  cholesterol  and  red  blood  cell  count).    Obvious  excep- 
tions to  this  are  urea  nitrogen  (all  methods) ,  uric  acid  (all  methods) , 
and  hemoglobin  (all  methods),  where  the  average  differences  are  small. 

As  with  the  interstate  group,  the  volunteer  laboratories  are 
generally  becoming  more  homogeneous.     In  fact,  for  seven  constituents, 
the  standard  deviation  for  volunteer  laboratories  was  smaller  after  eight 
or  nine  quarters  than  the  comparable  interstate  statistic.  Considering 
that  the  volunteer  laboratory  standard  deviation  was  initially  larger  for 
10  out  of  the  13  constituents,  the  group  as  a  whole  has  made  greater 
strides  towards  increased  homogeneity  than  have  the  interstate  laboratories. 


4.2.    Profile  Information:    Licensed  Laboratories  Only 

The  licensed  laboratories  were  characteri2ed  according  to  several 
attributes  reported  to  CDC  by  the  laboratories.    For  instance,  it  was 
possible  to  determine  those  licensed  laboratories  which  are  involved  in 
one  or  more  additional  accreditation  or  licensure  programs,  the  number  of 
employees,  supervisor's  educational  achievements,  and  other  characteristics. 
Paragraphs  4.2.1  through  4.2.5  discuss  the  various  characteristics  of  the 
237  licensed  laboratories  for  which  data  was  available. 


4.2.1.    Additional  Program  Involvement 

Of  the  237  licensed  laboratories  for  which  data  was  available,  12 
percent  (29)  do  not  participate  in  any  other  inspection,  accreditation  or 
licensure  programs.    The  208  laboratories  that  do  participate  in  other 
programs  are  distributed  as  follows: 


Number 

Percent 

Participate  in  one  additional  program 

68 

29  • 

Participate  in  two  additional  programs 

93 

39 

Participate  in  three  additional  programs 

47 

20 

Subtotal 

208 

88 

Do  not  participate  in  other  programs 

29 

12 

Total 

237 

1001 

4.2.2.    Number  of  Full-Time  Employees 

More  than  half  of  the  237  laboratories  have  less  than  eight  full-time 
employees;  18  percent,  or  42  laboratories,  have  more  than  20  en^loyees. 
The  frequency  distribution  below  describes  this  characteristic: 
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Number 

Percent 

Laboratories  with  3  or  less  emnlovees 

Laboratories  with  4-7  employees 

75 

35 

Laboratories  with  8-11  employees 

33 

14 

Laboratories  with  12-20  employees 

34 

14 

Laboratories  with  more  than  20  employees 

42 

18 

Total 

237 

1001 

4.2.3.    Number  of  Supervisors  per  Laboratory 

Six  percent  of  the  licensed  laboratories  do  not  have  any  supervisors , 
while  71  percent  of  the  laboratories  have  1-4  supervisors.    The  breakdown 
is  as  follows : 


Number 

Percent 

Laboratories  with 

no  supervisors 

15 

6 

Laboratories  with 

1  supervisor 

100 

43 

Laboratories  with 

2  supervisors 

46 

19 

Laboratories  with 

3  or  4  supervisors 

46 

19 

Laboratories  with 

5-12  supervisors 

24 

10 

Laboratories  with 

more  than  12  supervisors 

6 

3 

Total 

237 

100% 

4.2.4.    Supervisor  Academic  Achievement 

There  were  620  individuals  identified  as  working  supervisors  in  the 
237  laboratories.    Of  these  620  supervisors,  roughly  17  percent  have  no 
college  degree.    The  following  table  contains  the  overall  frequency  of 
supervisors  having  attained  various  levels  of  academic  achievement. 


Number 

Percent 

Supervisors  with  no  college  degree 

106 

17 

Supervisors  with  a  B.S.  or  A.B.  Degree 

259 

42 

Supervisors  with  an  M.S.  or  M.A.  Degree 

53 

9 

Supervisors  with  a  Ph.D.  Degree 

72 

11 

Supervisors  with  an  M.D.  Degree 

80 

13 

Supervisors  with  other  degrees 

50 

8 

Total 

620 

100% 
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4.2.5.    Academic  Achievement  of  Supervisors  Within  Laboratories 

A  cross -tabulation  by  academic  achievement  and  number  of  supervisors 
per  laboratory  was  also  prepared;  that  is,  how  the  academic  degrees  are 
distributed  among  laboratories  with  different  numbers  of  supervisors. 
The  breakdowns  were  obtained  for  laboratories  with  one,  two,  and  three 
supervisors.    These  tabulations  follow: 

Laboratories  with  One  Supervisor 


Number  Percent 

Supervisors  with  no  college  degree  19  19 

Supervisors  with  a  B.S.  or  A.B.  Degree  46  46 

Supervisors  with  an  M.S.  or  M.A.  Degree  9  9 

Supervisors  with  a  Ph.D.  Degree  5  5 

Supervisors  with  an  M.D.  Degree  14  14 

Supervisors  with  other  degrees  7  7 


Total  100  100% 

Laboratories  with  Two  Supervisors 
First  Supervisor  Second  Supervisor  Frequency 

No  college  degree  No  college  degree  6 

No  college  degree  B.S.  or  A.B.  9 

No  college  degree  M.D.  1 

No  college  degree  Other  1 

B.S.  or  A.B.  B.S.  or  A.B.  9 

B.S.  or  A.B.  M.S.  or  M.A.  2 

B.S.  or  A.B.  Ph.D.  4 

B.S.  or  A.B.  Other  1 

M.S.  or  M.A.  Ph.D.  1 

M.S.  or  M.A.  M.D.  1 

M. S.  or  M.A.  Other  '  1 

Ph.D.  Ph.D^  1 

Ph.D.  M.D.  2 

Ph.D.  Other  1 

M.D.  M.D.  5 

Other  Other  2 

47 

Laboratories  with  Three  Supervisors 
First  Supervisor    Second  Supervisor    Third  Supervisor  Frequency 

None  B.S.  or  A.B.  M.S.  or  M.A.  2 

None  B.S.  or  A.B.  M.D.  1 

None  B.S.  or  A.B.  Other  2 
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Laboratories  with  Three  Supervisors  (continued) 
First  Supervisor    Second  Supervisor    Third  Supervisor  Frequency 


INUliC 

uuier 

1 

■R         n-r  A  "R 

Dm  O.     Ui    rV.  £>. 

"R  S     n-r  A  R 

Jj.  O.      UI     JAm  Dm 

None 

1 
1 

■R  9     o-r   A  R 

Dm  Jm       or     A.  X3. 

■7 
J 

■R         nr  A  R 

Dm  0«       Ul      S\m  Dm 

■R  9     r,-r  A  R 

M  n 

M .  XJ  . 

1 

B  S     nr  A  R 

R  <^     n-r  A  R 

iJ.  O  .      UX     I\m  Dm 

ULXiei 

X 

MS     nr  M  A 

Ph  Ti 

M  n 

M  .  XJ  . 

1 
1 

M.S.  or  M.A. 

M.S.  or  M.A. 

B.S.  or  A.B. 

1 

M.S.  or  M.A. 

M.S.  or  M.A. 

Ph.D. 

1 

r^ii.  D  m 

X  XI  mU  m 

i:^n.  XJ . 

1 

Ph.D. 

Ph.D. 

M.S.  or  M.A. 

1 

M.D. 

M.D. 

M.D. 

3 

M.D. 

.  M.D. 

B.S.  or  A.B. 

1 

M.D. 

M.D. 

Ph.D. 

3 

26 

All  other  combinations  had  a  zero  frequency;  i.e.,  in  none  of  the 
licensed  laboratories  that  had  three  supervisors  were  all  three  without 
any  college  degree. 


4.3.    Analyses  of  Variance 

A  one-way  analysis  of  variance  of  differences  among  test  results 
associated  with  each  of  the  independent  variables  (excluding  number  of 
employees)  was  performed  for  every  constituent,  using  the  NBS  OMNITAB  II 
computer  program.    Specifically,  the  values  of  relative  differences  for 
all  laboratories  and  all  time  periods  were  classified  according  to  one 
of  the  independent  variables,  and  the  differences  among  the  means  for 
this  classification  were  investigated.    The  results  of  these  analyses  are 
summarized  in  Paragraphs  4.3.1  through  4.3.4. 


4.3.1.  Accreditation 

The  extent  to  which  interstate  laboratories  participated  in  other 
accreditation  or  licensure  programs  did  not  significantly  (tested  at  the 
. 05  level  of  probability)  affect  their  average  performance  in  the  CDC 
proficiency  testing  program.    That  is,  the  average  difference  for  labora- 
tories not  involved  with  any  other  program  was  not  significantly  different 
from  a  comparable  value  associated  with  laboratories  involved  with  several 
other  programs.    This  finding  was  consistent  for  all  14  constituents. 


4.3.2.    State  Program  Intensity 

Analysis  of  variance  indicates  that  performance  in  the  CDC  profi- 
ciency testing  program  is  not  functionally  related  to  intensity  of  programs 

at  the  state  level,  for  all  14  constituents. 
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4.3.3.    Supervisory  Education 

Different  levels  of  education  achieved  by  the  working  supervisors 
did  not  effect  the  performance  of  licensed  laboratories  in  the  proficiency 
testing  program.    For  all  constituehts ,  there  was  no  indication  that 
laboratories  whose  work  is  supervised  by  persons  with  Ph.D.  degrees 
performed  at  different  levels  than  their  counterparts  whose  supervisors 
had  baccalaureate  degrees  or  no  degree  at  all. 


4.3.4.    Analytical  Method/Laboratories 

The  data  indicate  that  interstate  laboratory  performance  in  the  CDC 
proficiency  testing  program  is  significantly  (p  <  .001)  affected  by  the 
analytical  method  used.    This  finding  was  consistent  for  all  14  constit- 
uents.   However,  the  laboratories  included  in  the  study  were  not  assigned 
at  random  to  the  various  methods,  and  each  laboratory  used  one  and  only 
one  method  to  analyze  each  constituent.    Therefore,  unless  one  is  willing 
to  make  the  necessary  assumption  that  interstate  laboratories  are  a 
homogeneous  group,  and  that  the  differences  observed  are  truly  due  to  the 
analytical  methods  alone  and  not  to  factors  already  analyzed,  there  are 
no  firm  conclusions  which  can  be  drawn  from  this  simple  one-way  analysis 
of  variance.    Additional  analyses  of  variance,  which  consider  the  possible 
interactions  of  the  four  independent  variables,  will  be  conducted  in  the 
near  future  and  reported  separately.    Appendix  E  consists  of  summary 
tables  developed  from  this  one-way  analysis  of  variance. 


4.4.    Method  Bias  in  Determining  Target  Values 

An  original  premise  underlying  this  entire  evaluative  effort  was 
that  one  measure  of  a  laboratory's  analytical  performance  is  its  ability 
to  come  close  to  the  "right  answer."    In  this  study  the  "right  answer," 
for  each  sample,  was  considered  to  be  the  average  analytical  value 
reported  by  a  group  of  at  least  30  reference  laboratories,  all  using  the 
same  analytical  method  as  the  licensed  laboratory.    The  limited  number  of 
cases  where  at  least  30  laboratories  used  the  same  method  restricted  the 
number  of  method  comparisons  which  could  be  made.    Thus,  the  available 
data  base  was  insufficient  to  conclusively  analyze  the  importance  of 
establishing  target  values  on  a  method-by-method  basis.  Nevertheless, 
the  limited  results  were  suggestive.    Table  4.1  summarizes  the  number  of 
times  the  method  comparisons  were  possible  for  each  constituent. 

The  tests  of  significance,  which  were  performed  on  the  means  (t 
test  with  unequal  sample  size,  known  variability)  of  the  groups,  did  not 
indicate  any  statistical  significance,  at  the  10  percent  level,  between 
the  methods  compared. 
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Table  4.1.    Summary  of  Number  of  Times  Method  Comparisons 
Were  Possible  for  Each  Constituent 


Constituents  Analytical  Methods  Number  of  Method 
   Comparisons 

Glucose                AutoAnalyzer  vs  SMA.  3 

Creatinine  Alkaline  Picrate  (w/o  Lloyds)  vs 

AutoAnalyzer  5 

Urea  Nitrogen      AutoAnalyzer  ys  3AA  6 

Uric  Acid            AutoAnalyzer  ys  SMA  6 

Uric  Acid  Phosphotungstate  (carbonate)  vs 

AutoAnalyzer  3 

Uric  Acid  Phosphotungstate  (carbonate)  vs 

SMA  3 

Total  Protein       Biuret  (serum)  ys_  Refractive  index  3 

Total  Protein       Biuret  (serum)  ys  AutoAnalyzer  3 

Total  Protein       Biuret  (serum)  ys_  S^A  3 

Total  Protein       Refractive  index  ys_  AutoAnalyzer  3 

Total  Protein      Refractive  index  ys_  SMA  3 

Cholesterol  Direct  serum  (acetic  anhydride)  vs 

AutoAnalyzer  3 

Cholesterol  Direct  serum  (acetic  anhydride)  ys^ 

SMA  3 

Cholesterol          AutoAnalyzer  ys_  avIA  3 

Calcium               Atomic  absorption  ys  AutoAnalyzer  3 

Calcium               Atomic  absorption  ys_  SVIA  3 

Calcium               AutoAnalyzer  ys  3^  3 

Sodium       /         Flame  photometry  vs  AutoAnalyzer  8 

Sodium                 Flame  photometry  ys  S^IA  3 

Potassium  ■  .  ..        Flame  photometry  ys  AutoAnalyzer  11 

Potassium             Flame  photometry  ys_  SMA  3 

Potassium            AutoAnalyzer  ys  SAk  3 

Chloride              Cotlove  titrator  vs  AutoAnalyzer  3 

Chloride              Cotlove  titrator  ys  SMA  3 

Chloride              AutoAnalyzer  ys_  SVIA  3 

Hemoglobin  Cyanmethemoglobin  (manual)  ys 

Coulter  S  3 
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5.    Summary  and  Conclusions 


5.1.  Summary 

This  report  summarizes  the  first  part  of  a  two-part  study  conducted 
for  the  Division  of  Health  Evaluation,  Office  of  the  Assistant  Secretary 
for  Planning  and  Evaluation,  Department  of  Health,  Education,  and  Welfare. 
The  study  consisted  of  an  examination  of  data  from  the  chemistry  and 
hematology  portions  of  the  first  two  years  of  a  proficiency  testing 
program  for  laboratories  licensed  under  the  Clinical  Laboratory  Improve- 
ment Act  of  1967.    Also  included  in  the  report  is  Appendix  A,  a  review 
of  the  legislative  history  behind  the  law, and  Appendix  B,  an  annotated 
bibliography  of  pertinent  literature  in  the  field  of  laboratory  profi- 
ciency testing  from  1963  to  1971. 

Statistical  analyses  were  performed  on  two  sets  of  data  supplied  by 
the  Center  for  Disease  Control.    One  set,  covering  268  licensed  clinical 
laboratories  and  several  hundred  volunteer  laboratories,  included  the 
results  of  proficiency  tests  conducted  in  these  blood  constituents : 
glucose,  creatinine,  urea  nitrogen,  uric  acid,  total  protein,  cholesterol, 
triglyceride,  calcium,  sodium,  potassium,  chloride,  hematocrit,  red  blood 
cell  count  and  hemoglobin.    The  second  set  of  data  contained  certain 
profile  information  on  237  of  the  268  licensed  laboratories.    This  infor- 
mation included  the  number  of  full-time  employees,  the  highest  academic 
achievement  level  of  the  working  supervisors,  a  measure  of  the  strength 
of  the  proficiency  testing  program  operated  by  the  state  in  which  the 
laboratory  was  located,  and  the  total  number  of  accreditation  or  licensure 
programs  in  which  the  laboratory  was  involved. 


5.2.  Conclusions 

It  is  very  important  to  keep  the  following  three  aspects  of  profi- 
ciency testing  programs  in  general,  and  the  CDC  program  in  particular, 
in  mind  when  reviewing  the  data  and  formulating  conclusions. 

1.  Almost  every  proficiency  testing  program  measures  the  capability 
of  participating  laboratories  instead  of  their  routine  level  of 
performance.    The  extent  to  which  capability  and  performance  are 
related  is  unknown. 

2.  Factors  such  as  scoring  methods,  criteria  for  determining  out- 
liers, and  inter- convertibility  among  methods  are  known  to 
affect  measured  performance  in  proficiency  testing  programs. 
Greater  standardization  among  the  major  testing  programs  would 
be  desirable. 

3.  Proficiency  testing  is  only  one  aspect  of  a  broad  based  quality 
assurance  program.    It  is  also  necessary  to  have  well  trained 
laboratory  personnel,  fresh  reagents,  standard  solutions  of 
high  quality,  and  timely  feedback  to  the  laboratory. 

The  following  conclusions  fall  into  two  major  categories.    The  first 
three  deal  specifically  with  the  data  analyses.    The  fourth  conclusion 
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is  identified  with  the  proficiency  testing  program  currently  being 
conducted  by  the  Center  for  Disease  Control. 


1.  The  only  independent  variable  for  vdiich  the  mean  differences 
were  significantly  different  was  Analytical  Method/Laboratories. 
The  other  three  variables,  Accreditation,  State  Program  Inten- 
sity, and  Supervisory  Education,  were  apparently  unimportant. 

2.  The  standard  deviation,  a  measure  of  the  variability  among 
laboratories ,  has  generally  decreased  over  the  two-year  period 
under  study  for  both  the  interstate  and  volunteer  laboratory 
groups.    This  is  an  indication  that  both  groups  have  become 
more  homogeneous  in  their  testing  of  proficiency  program 
samples . 

3.  The  data  do  not  support  establishing  reference  group  target 

i      values  on  a  method- by-method  basis.    More  data,  collected  in  a 
statistically  designed  experiment,  would  be  helpful  in  reaching 
more  rigorous  and  definitive  conclusions. 

4.  There  are  few  indications  of  meaningful  changes  in  the  average 
level  of  performance  of  laboratories  participating  in  the  CLIA 
proficiency  testing  program.    Consideration  should  be  given  to 
a  sampling  scheme  somewhat  akin  to  skip- lot  sampling  for  those 
constituents  which  appear  to  present  no  analytical  challenge  to 
laboratories.    Greater  emphasis  might  then  be  placed  on  those 
constituents  which  give  licensed  laboratories  the  most  diffi- 
culty (cholesterol  and  creatinine  for  example) . 
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Appendix  A.    A  Review  of  tJie  Legislative  History  of  CLIA  '67 


Introduction 

A  review  of  the  pertinent  legislative  material,  including  House  and 
Senate  report  hearings  and  Federal  Register  (FR)  announcements  was  made 
in  search  of  answers  to  questions  such  as: 

1.  At  the  time  of  the  hearings,  what  was  the  projected  magnitude 
of  the  licensing  effort?    Approximately  how  many  clinical 
laboratories  were  engaged  in  interstate  commerce? 

2.  What  were  the  ostensible  problems  in  the  clinical  laboratory 
industry  which  led  to  the  hearings  and  ultimate  legislation? 


Public  Law  and  Rule  Making 

The  Honorable  Harley  0.  Staggers,  Chairman  of  the  House  Committee  on 
Interstate  and  Foreign  Commerce,  introduced  HR  6418  early  in  1967,  at  the 
request  of  the  Administration.    A  similar  version,  S  724,  was  introduced 
in  the  Senate,  and  the  bill  which  was  eventually  enacted  was  entitled 
"Clinical  Laboratory  Improvement  Act  of  1967,  Section  353  of  the  Public 
Health  Service  Act."    The  intent  of  the  Administration  was  to  extend  and 
strengthen  the  Partnership  for  Health  Provision,  Public  Health  Service 
Act,  PL  89-749,  by  placing  primary  reliance  upon  state  health  authorities 
and  their  services  programs.    Broadly  stated,  the  administration  viewed 
the  partnership  as  one  where  the  Federal  government  would  assist  the 
states  with  financial  and  technical  resources  without  assuming  any 
authority  for  the  conduct  of  the  state  programs. 

Three  major  witnesses  for  the  administration  were  Undersecretary  of 
HEW,  Wilbur  J.  Cohen;  Surgeon  General,  Dr.  William  H.  Stewart;  and 
Communicable  Disease  Center  Director,  Dr.  David  J.  Sencer.    Dr.  Sencer 
testified  that  25  percent  of  all  clinical  laboratory  analyses  were  so 
gross  as  to  be  medically  useless.    This  testimony,  from  so  eminent  a 
witness,  made  a  significant  impact  upon  the  congressional  committee 
before  which  Dr.  Sencer  was  appearing. 

The  major  mileposts  with  respect  to  publishing  the  new  legislation 
and  its  projected  impact  on  the  industry  were: 

1.  October  15,  1968 — A  notice  of  rule  making  was  published  in  the 
Federal  Register  (33  FR  15297-15303)  which  proposed  the  addition 
of  a  new  Part  74  to  the  Public  Health  Service  Regulations. 

2.  December  31,  1968 — A  notice  that  the  new  Part  74  was  added  to 
the  PHS  Regulations,  effective  30  days  after  publication  (Jan- 
uary 31,  1969).    This  notice  described  the  nine  major  subparts 
of  the  new  Part  74  as  follows: 

Subpart  A  -  Definitions  and  Applicability 
Subpart  B  -  License:    Application  and  Issuance 
Subpart  C  -  Quality  Control 
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Subpart  D  -  Personnel  Standards 
Subpart  E  -  Proficiency  Testing 

Subpart  F  -  Accreditation — State  Loans — Stringency  of 

Standards — Termination 
Subpart  G  -  General  Provisions 

Subpart  H  -  Revocation,  Suspension,  and  Limitation  of 
Licenses  and  Letters  of  Exemption  Notice 
Subpart  I  -  Hearings  (Reserved) 

3.  April  5,  1969 — Effective  date  of  the  Act  postponed  again  to 
July  1,  1969  (it  had  already  been  postponed  from  January  31  to 
April  1,  1969  in  paragraph  2,  Subpart  I  of  FR  No.  253). 

4.  July  1,  1969 — The  College  of  American  Pathologist  proficiency 
testing  program  was  determined  to  be  equal  to  or  more  stringent 
than  the  Federal  program.    Laboratories  enrolled  in  the  CAP 

■      program  were  granted  exemption  from  CLIA  upon  submitting  an 
application  for  same. 

5.  August  15,  1969 — Effective  date  for  cytotechnologists  to  become 
qualified  extended  from  January  1,  1960  to  January  1,  1969. 

6.  October  14,  1970 — Notice  in  the  FR  that  DHEW  proposed  to  amend 
the  CLIA  '67  regulations  with  regard  to  granting  hearings  for 
affected  laboratories. 

7.  October  25,  1969 — HEW  published  a  notice  of  intent  in  the  FR  to 
require  a  letter  of  exemption  from  laboratories  accepting  fewer 
than  100  specimens  per  year  in  a  specific  category  or  subcategory. 

8.  December  3,  1969 — A  notice  published  in  the  FR  to  amend  CDC's 
Statement  of  Organization,  Functions,  and  Delegation  of  Authority. 

9.  February  7,  1970 — A  list  of  laboratories  which  were  granted 
exemption  from  CLIA  '67  was  published  by  HEW  in  the  FR. 

10.    April  16,  1970— 4IEW  published  a  notice  in  the  FR  that  item  7 
above  was  now  in  effect;  exempt  laboratories  would  have  to 
have  a  letter  of  exemption,  issued  by  the  director,  CDC. 

Licensure — Eligibility  and  Exemption 

There  has  never  been  a  complete  census  of  all  operating  clinical 
laboratories  in  the  United  States.    Such  an  undertalcing  would  be  facili- 
tated by  a  consensus  as  to  a  working  definition  of  "a  clinical  laboratory." 
Current  and  past  estimates  of  the  total  number  of  laboratories  range  from 
15,000  to  perhaps  200,000;  depending  upon  whether  physicians'  offices  are 
included. 

When  the  director  of  the  Center  for  Disease  Control  (CDC)  testified 
at  the  House  hearings,  he  estimated  that  "Ten  percent  of  the  clinical 
laboratories  in  this  country  are  in  interstate  commerce."    There  are 
other  references  to  the  number  of  laboratories  in  interstate  commerce, 
such  as  that  which  appears  on  page  2083  of  the  Congressional  and  Admin- 
istrative News,  90th  Congress  (First  Session,  1967):    "Estijnates  given  to 
the  Committee  by  the  Department  of  Health,  Education  and  Welfare  indicate 
that  there  are  approximately  1,000  laboratories  substantially  operating 
in  interstate  commerce.    These  are  primarily  independent  laboratories 
and  a  few  hospital  laboratories."    The  Administration's  intent,  in 
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supporting  this  legislation,  was  to  impact  on  the  largest  segment  of  the 
industry  as  evidenced  by  the  following  quotation  from  the  same  90th 
Session  Congressional  and  Administrative  News:    "The  committee  believes 
that  the  Secretary  of  Health,  Education, and  Welfare  should  be  required 
to  provide  an  exemption  from  licensing  in  those  cases  where  laboratory 
operations  in  commerce  are  so  small  or  infrequent  as  not  to  constitute 
a  significant  threat  to  the  public  health." 

There  were  many  forces  acting  on  the  Congress  during  its  considera- 
tion of  the  legislation,  such  as  the  life  insurance  industry  which  wanted 
exemption  for  laboratories  whose  primary  business  was  associated  with 
performing  tests  in  conjunction  with  physical  examinations  for  policy 
holders,  the  medical  community  which  wanted  any  physician  directed  labora- 
tory to  be  exempt  from  federal  intervention,  and  various  professional 
(society)  accreditation  organizations  which  operate  financially  success- 
ful quality  control  programs.    Consequently,  when  Public  Law  90-174, 
"The  Partnership  for  Health  Amendments  of  1967"  was  enacted  on  December 
5,  1967,  it  contained  the  following  exemptions  on  page  580  of  the  U.S. 
Code — 

(2)  A  license  issued  under  this  section  shall  be  valid  for  a  period 
of  three  years,  or  such  shorter  period  as  the  Secretary  may  estab- 
lish for  any  clinical  laboratory  or  any  class  or  classes  thereof; 
and  may  be  renewed  in  such  manner  as  the  Secretary  may  prescribe. 
The  provisions  of  this  section  requiring  licensing  shall  not  apply 
to  a  clinical  laboratory  in  a  hospital  accredited  by  the  Joint 
Commission  on  the  Accreditation  of  Hospitals  or  by  the  American 
Osteopathic  Association,  or  a  laboratory  which  has  been  inspected 
and  accredited  by  such  commission  or  association,  by  the  Commission 
on  Inspection  and  Accreditation  of  the  College  of  American  Patholo- 
gists, or  by  any  other  national  accreditation  body  approved  for  the 
purpose  by  the  Secretary,  but  only  if  the  standards  applied  by  such 
Commission,  association,  or  other  body  in  determining  whether  or 
not  to  accredit  such  hospital  or  laboratory  are  equal  to  or  more 
stringent  than  the  provisions  of  this  section  and  the  rules  and 
regulations  issued  under  this  section,  and  only  if  there  is  adequate 
provision  for  assuring  that  such  standards  continue  to  be  met  by 
such  hospital  or  laboratory;  provided  that  any  such  laboratory  shall 
be  treated  as  a  licensed  laboratory  for  all  other  purposes  of  this 
section. 

and  on  page  582 — 

(i)  The  provisions  of  this  section  shall  not  apply  to  any  clinical 
laboratory  operated  by  a  licensed  physician,  osteopath,  dentist, 
or  podiatrist,  or  group  thereof,  who  performs  or  perform  laboratory 
tests  or  procedures,  personally  or  through  his  or  their  employees, 
solely  as  an  adjunct  to  the  treatment  of  his  or  their  own  patients; 
nor  shall  such  provisions  apply  to  any  laboratory  with  respect  to 
tests  or  other  procedures  made  by  it  for  any  person  engaged  in  the 
business  of  insurance  if  made  solely  for  purposes  of  determining 
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whether  to  write  an  insurance  contract  or  of  determining  eligi- 
bility or  continued  eligibility  for  payments  thereunder. 

(1)  Where  a  State  has  enacted  or  hereafter  enacts  laws  relating  to 
matters  covered  by  this  section,  which  provide  for  standards  equal 
I      to  or  more  stringent  than  the  provisions  of  this  section  or  than 
the  rules  and  regulations  issued  under  this  section,  the  Secretary 
may  exempt  clinical  laboratories  in  that  State  from  compliance  with 
this  section. 

As  of  January  18,  1972,  there  were  552  clinical  laboratories 
licensed  by  the  Center  for  Disease  Control  out  of  approximately  1,500 
which  are  "probably"  eligible  from  the  standpoint  of  being  engaged  in 
interstate  commerce.    The  63  percent  not  licensed  are  assumed  to  be 
exempt  because  they  accept  less  than  100  samples  per  year  in  interstate 
commerce  or  are  otherwise  accredited  or  licensed. 


Summary 

The  committee  hearings  and  related  evidence  indicated  that  the 
performance  of  clinical  laboratories  was  not  what  it  could  or  should  be. 
The  committee  recognized  that  improvement  of  the  clinical  laboratories 
and  their  performance  was  a  long  run  effort  for  the  state  governments, 
their  health  agencies  and  their  public  health  service  authorities.  How- 
ever, the  committee  also  recognized  that  it  would  take  time  for  the  state 
licensing  programs  to  be  developed  throughout  the  nation.    In  the  mean- 
time, the  state  programs  would  vary  considerably  with  respect  to  proce- 
dures and  quality.    Because  of  this  variance,  laboratories  doing  business 
in  interstate  commerce  could  locate  in  states  with  weaker  licensing 
programs  or  in  states  without  programs.    In  addition  to  the  need  for 
licensing  interstate  business  of  the  clinical  laboratories,  there  was 
the  above-mentioned  administration  and  congressional  plan  to  extend  and 
strengthen  the  partnership  for  health  provision  of  the  PHS  Act.  These 
elements,  the  need  for  regulations  of  interstate  samples  and  the  exten- 
tion  of  the  partnership  for  health,  were  the  motivating  factors  for 
CLIA  '67.    However,  an  unforeseen  development  in  certain  states  has, 
to  some  extent,  bypassed  the  anticipated  effect  of  the  law.    At  least 
two  states  have  passed  laws  which  prohibit  solicitation  of  laboratory 
work  from  laboratories  outside  that  state. 
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Appendix  B.  Abstracts 


The  entries  in  this  annotated  bibliography  on  proficiency  testing 
programs  in  the  open  literature  have  been  obtained  from  a  search  of  the 
Department  of  Commerce  National  Technical  Information  Center;  the  Defense 
Documentation  Center;  the  Index  Medicus  of  the  National  Library  of  Medi- 
cine, DHEW  library;  from  references  in  the  papers  themselves,  and  from 
a  report  entitled  "Clinical  Laboratory  Evaluation  Programs"  dated  July 
31,  1970,  by  the  Auerbach  Corporation,  Philadelphia.    This  last  report 
was  procured  by  the  Department  of  Health,  Education  and  Welfare  on 
Contract  110-69-434. 

The  34  entries  on  proficiency  programs  are  arranged  by  years  start- 
ing with  1963,  and  are  alphabetical  within  years. 


1963 

1.  Tonks,  D.  B. ,  "A  Study  of  the  Accuracy  and  Precision  of  Clinical 
Chemistry  Determinations  in  170  Canadian  Laboratories,"  Clinical 
Chemistry,  9:217  (1963). 

The  data  obtained  have  shown  that  there  was  a  lack  of  both  accuracy 
and  precision  in  many  laboratories.    There  is  obviously  a  need  for  an 
improvement  in  the  methods  of  detecting  errors  and  of  evaluating  p  ci- 
sion  in  the  laboratory.    An  adequate  quality  control  program  and  the 
proper  usage  of  control  serums  are  indispensable  for  these  purposes.  It 
is  also  useful  for  every  laboratory  to  take  part  in  periodic  studies  such 
as  this  one  in  order  to  obtain  an  unbiased  evaluation  of  its  performance. 


1967 

1.  Copeland,  B.  F. ,  W.  J.  Blake,  R.  J.  Muelling,  and  L.  P.  Skendzel, 
"A  Report  of  the  Standards  Committee  of  the  College  of  American 
Pathologists,"  American  Journal  of  Clinical  Pathology,  48:104  (1967). 

The  purpose  of  this  survey  was  to  stimulate  interest  in  the  accuracy 
and  precision  of  chemical  measurements  commonly  made  in  laboratories  for 
the  diagnosis  and  therapy  of  disease.    The  following  measurements  were 
included  in  the  survey:    glucose,  urea  nitrogen,  bilirubin,  cholesterol, 
Calcium,  phosphorous,  sodium,  potassium,  bicarbonate,  uric  acid,  and 
chloride . 

2.  Kaufmann,  William  and  R.  Vanderlinde,  "Medical  Laboratory  Evaluation," 
New  England  Journal  of  Medicine,  277:1024  (1967). 

The  overwhelming  evidence  points  to  rather  serious  deficiencies  in 
the  performance  of  tests,  in  their  reproducibility  and  thus  in  their 
accuracy  and  dependability  as  aids  in  clinical  diagnosis.    A  built-in 
fallacy  in  assessing  laboratory  performance  is  the  assumption  that  the 
test  specimen  is  treated  in  the  same  fashion  as  the  daily  routine  specimen 
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1967  (continued) 


whereas  this,  in  fact,  rarely  happens.    The  data  show  that  low  quality 
in  testing  is  related  to  a  low  annual  number  of  tests  performed.  The 
fact  that  there  is  a  serious  laboratory  problem  in  this  country  should 
be  recognized. 

3.    Schaeffer,  M. ,  D.  Widdock,  S.  Blatt,  and  M.  E.  Wilson,  "The  Clinical 
Laboratory  Improvement  Program  in  New  York  City:    I.    Methods  of 
Evaluation  and  Results  of  Performance  Tests,"  Health  Laboratory 
Science,  4:72  (1967). 

This  report  is  limited  to  the  data  obtained  in  evaluating  the 
capabilities  of  all  the  laboratories  within  the  jurisdiction  of  New  York 
City  and  to  factors  that  relate  to  the  quality  of  performance. 

In  October  1966  there  were  359  clinical  laboratories  with  permits  to 
operate  in  New  York  City.    A  different  group  of  laboratories  was  tested 
each  week,  and  each  laboratory  was  tested  at  least  four  times  in  a 
two-year  period.    The  tests  performed  included  microbiology,  syphilis 
serology,  cross-matching  and  hemoglobin  determinations.    The  standard 
deviation  of  all  the  results  of  the  reference  laboratories  was  calcu- 
lated, the  acceptable  range  being  two  standard  deviations  from  the  mean. 
About  50%  of  the  independent  laboratories  and  about  40%  of  all  of  the 
hospital  laboratories  were  rated  as  unsatisfactory  in  bacteriology,  while 
about  25-30%  were  rated  as  excellent.    In  clinical  chemistry,  about  50% 
were  fair,  and  the  remainder  equally  divided  between  unsatisfactory  and 
excellent.    The  poorest  results  appear  among  the  independent  laboratories, 
although  the  other  laboratories  were  not  much  better  off.    About  18%  of 
the  cross -matching  tests  were  unsatisfactory;  in  syphilis  serology  84% 
of  the  results  of  independent  laboratories  and  more  than  90%  of  the  results 
of  hospital  laboratories  were  excellent. 

Laboratory  performance  did  not  appear  to  be  related  to  whether  the 
director  was  full-time  or  not.    Qualifications  of  the  director  did  not 
appear  to  matter.    The  organization  of  the  laboratory  did  appear  to  in- 
fluence performance.    Higher  test  accuracy  was  noted  in  the  laboratories 
that  processed  high  volumes  of  specimens. 

There  appeared  to  be  a  direct  relationship  between  the  use  of 
professionally  trained  supervisors  and  the  quality  of  performance.  Forty 
excellent  laboratories,  representing  only  29.6%  of  voluntary  hospitals, 
employed  63.3%  of  all  the  supervisors  in  voluntary  hospitals.    This  level 
of  supervision  is  difficult  to  obtain  and  too  expensive  for  a  small 
laboratory.    This  paper  expresses  the  opinion  that  the  existing  situation 
(1964-1966)  in  clinical  laboratories  cannot  be  permitted  to  continue 
indefinitely. 
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1.  Barnett,  Roy  N. ,  "Medical  Significance  of  Laboratory  Results," 
American  Journal  of  Clinical  Pathology,  50:671  (1968). 

Tests  performed  in  clinical  laboratories  are  among  the  main 
diagnostic  aids  available  to  physicians.    The  great  demand  for  these 
tests  and  the  enormous  number  performed  have  led  to  Federal  specifica- 
tion through  the  Medicare  Act  and  the  Clinical  Laboratory  Improvement 
Act  of  1967  (CLIA  '67).    Regulations  under  both  Acts  set  basic  standards 
of  operation  and  require  participation  in  a  proficiency  testing  program. 

The  basic  question  is  how  accurate  must  clinical  laboratory  work  be? 
The  Standards  Committee  of  the  College  of  American  Pathologists  has  been 
the  major  proficiency  surveying  body  in  the  United  States  for  a  number  of 
years,  and  these  surveys  lead  to  two  questions:     (1)  How  well  are  labora- 
tories perfoming  certain  common  analyses?  and  (2)  Is  this  kind  of 
performance  adequate  for  good  medical  practice?    This  provisional  report 
examines  these  questions.    Six  (sic)  terms  are  defined,  including  accuracy, 
precision,  standard  deviation,  coefficient  of  variation,  percentiles, 
normal  range,  medically  significant,  state-of-the-art,  and  decision  level. 
Other  topics  include  guidelines  in  establishing  limits  for  reporting 
values  of  medical  significance,  a  table  of  medically  significant  values, 
interlaboratory  and  intralaboratory  evaluation,  and  the  utilization  of 
specific  limits  in  medical  practice. 

2.  College  of  American  Pathologists,  "Guidelines  for  Evaluating 
Laboratory  Performance  in  Survey  and  Proficiency  Testing  Programs," 
American  Journal  of  Clinical  Pathology,  49:457  (1968). 

Yearly  surveys  have  produced  data  that  describe  the  accuracy  and 
reproducibility  of  procedures.    Variability  increases  as  variables  are 
introduced.    Single  laboratory  precision  is  better  than  multiple  labora- 
tory precision. 

In  consultation  with  Dr.  W.  J.  Youden,  NBS,  the  CAP  established 
criteria  for  acceptability  in  interlaboratory  surveys,  but  not  for  pre- 
cision objectives  by  individual  laboratories.    Reference  laboratory 
results  will  be  provided  to  survey  participants.    Performance  terminology 
is  given,  as  are  good  performance  criteria  for  hemoglobin,  glucose,  urea 
nitrogen,  calcium,  phosphorous,  total  protein,  chloride,  uric  acid, 
albumin,  bilirubin,  sodium,  potassium,  creatinine,  and  cholesterol. 

3.  Martinek,  R.  G.,  et  al.,  "Comparison  of  Methods  for  Conducting 
Laboratory  Proficiency  Surveys,"  Health  Laboratory  Science,  5:239- 
256  (1968). 

The  program  reported  here  was  planned  and  carried  out  by  the  Illinois 
State  Department  of  Public  Health.    The  survey  method  used,  whereby 
unknowns  are  analyzed  by  participating  laboratories,  is  reported  and 
summarized  in  the  report.    Another  method  relied  on  the  use  of  patients' 
data  (PD) ,  wherein  each  laboratory  was  instructed  to  list  90  consecutively 
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determined  values.    These  distributions  were  plotted,  and  normal  curves 
fitted,  each  being  described  by  the  mean  or  average  and  the  corresponding 
standard  deviation.    Normal  ranges  were  established  for  each  test  used. 
From  the  data  from  the  conventional  surveys  by  the  American  Association 
of  Blood  Bariks  and  CAP  correlation  analyses  were  made  as  well  as  studies 
of  within  and  between  surveys  using  Kendall's  coefficient  of  rank 
correlation.    Results  "...indicated  as  close  a  relationship  between  a 
proficiency  sample  and  the  patients'  means  as  between  proficiency  samples 
from  the  two  agencies."    Considerable  clarification  of  interpretation 
was  obtained.    "When  the  conventional  proficiency  survey  results  are 
interpreted  in  this  manner  the  conclusion  is  inescapable  that  the  patients' 
data  is  informative  and  directly  related  to  our  concern."    The  results  of 
testing  patients'  specimens  represents  everything  that  the  laboratory  does. 
The  analyses  of  patients'  data  yield  standard  deviations  that  are  similar 
in  magnitude  to  those  from  conventional  and  extensive  surveys  of  labora- 
tories for  prothrombin  time.    Conclusions  from  these  analyses  are  (1) 
do  not  use  either  technique  as  the  basis  for  clinical  laboratory  regula- 
tory action  at  this  time,  and  (2)  carefully  evaluate  the  various  quality 
control  methods. 

4.  Prier,  J.  E. ,  L.  Sideman  and  I.  J.  Yankevitch,  "Clinical  Laboratory 
Proficiency  Testing,"  Health  Laboratory  Science,  5:12  (1968). 

Clear  evidence  documents  deficiencies  in  hospital  and  independent 
clinical  laboratories  (CL) .    Available  methodology  provides  ways  to 
obtain  high  accuracy  and  consistency  if  properly  applied.    CL  evaluation 
involves  three  factors:     (1)  personnel  qualifications,  (2)  on-site 
inspection,  and  (3)  proficiency  testing.    Proficiency  testing  (PT)  in 
Pennsylvania  was  made  mandatory  in  1961.    Two  responses  to  law  are  noted: 
(a)  well  trained  pathologists  provide  the  least  objections;  (b)  "notable" 
objections  are  from  directors  who  are  marginal  in  both  training  and 
facilities.    PT  must  be  carried  out  to  assure  competency  of  CL  work.  It 
is  doubtful  that  an  adequate  CL  control  program  can  be  obtained  by  volun- 
tary participation.     (Gives  history  of  PT  in  Pennsylvania.)    Any  effective 
program  must  be  enforced.    Ten  characteristics  of  a  new  PT  program  are 
discussed.    Competency  in  CL  work  must  include  PT,  and  PT  without  enforce- 
ment is  inadequate,  collecting  only  disquieting  data.    A  PT  program  must 
reach  all  laboratories  equally.    There  should  be  no  provision  for  waiver 
because  of  education  of  director  or  other  reasons.    Accurate  laboratory 
work  is  related  to  close  supervision.    Inspections  and  "on-the-spot" 
testing  are  essential  parts  of  a  total  evaluation  program.  Programs 
provided  by  legislation  with  provisions  for  compulsory  PT  are  mandatory. 

5.  Report  from  the  Ad  Hoc  Committee  on  "Documentation  of  Procedures, 
Diagnostic  Criteria,  and  Results  in  Laboratory  Medicine  of  the 
Committee  on  Pathology,"  Division  of  Medical  Science  NRC-NAS-NAE, 
American  Journal  of  Clinical  Pathology,  49:757-760  (1968). 

Conclusions:     (1)  That  there  is  a  need  for  documentation  of  medical 
laboratory  data  in  the  United  States  in  order  to  promote  the  comparability 

24 


196S  [continued) 


of  data  over  long  periods  and  bet^veen  different  sections  of  the  country 
as  patient  mobility  increases. 

(2)  That  the  questionnaire  method  has  been  successful  in  determining 
the  state-of-the-art  as  practiced  in  leading  medical  institutions. 

(3)  That  information  on  the  state-of-the-art  should  continue  to  be 
collected  and  should  be  published. 

(4)  That  a  Conference  on  Documentation  of  Procedures,  Diagnostic 
Criteria,  and  Results  in  Laboratory  Medicine  should  be  held. 

(5)  That  there  appears  to  be  a  definite  need  for  a  Center  for 
Documentation  of  Procedures,  Diagnostic  Criteria,  and  Results  in  Labora- 
tory Medicine.    The  obvious  benefit  to  patient  care  makes  this  an 
important  goal. 


1969 

1.    Eilers,  R.  J.,  "Total  Quality  Control  in  the  Medical  Laboratory," 
Hospital  Progress,  50:92  (1969). 

The  concept  of  total  quality  control  for  the  medical  laboratory  offers 
a  means  of  achieving  the  professional  objective  of  iinproving  the  quality 
of  the  service  that  it  renders.    Quality  Control  does  not  mean  best,  but 
it  could.    The  word  "control"  in  the  phrase  Quality  Control  includes  a 
four-part  management  tool : 

Establishment  of  standards 

Determination  of  conformance  to  these  standards 
Taking  corrective  action 
Planning  for  improved  standards. 

Nine  benefits  from  improved  Quality  Control  in  laboratories  include: 
Improvement  in  quality  of  laboratory  results 
Improvement  of  employee  morale 
Improvement  of  inspection  methods 
•Establishment  of  firm  time  standards 
Establishment  of  preventive  maintenance  schedules 
Development  of  a  factual  basis  for  cost  accounting. 

The  cost  of  obtaining  and  maintaining  a  level  of  quality  in  a 
laboratory  is  consolidated  with  the  cost  from  failure  to  obtain  the  given 
level  of  quality.    Four  elements  of  cost  are  cited:     (1)  Prevention  cost, 
(2)  Appraisal  costs,  (3)  Costs  of  internal  failure,  and  (A)  Costs  of 
external  failures.    A  total  Quality  Control  program  can  be  divided  into 
three  phases: 

Assign  quality  responsibility  to  all  employees 

Employ  a  quality  control  specialist 

Develop  special  techniques  for  all  laboratory  functions. 

Six  phases  of  Quality  Control  are: 
Xew  design  control 
Incoming  material  control 
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Process  control 
Production  output  control 
Product  reliability 
Special  process  studies. 

The  College  of  American  Pathologists  has  an  inspection  and  accredit- 
ation activity  for  total  laboratory  services  that  is  accepted  by  the  JCAH 
and  CDC.    This  service  has  been  in  operation  for  many  years.  Participants 
in  1968  numbered  3900. 

To  obtain  high  Quality  Control,  all  in  the  chain  must  recognize  the 
existing  deficiencies  and  initiate  programs  for  raising  the  existing 
level,  otherwise  responsibility  for  this  function — Quality  Control — will 
become  the  responsibility  of  state  or  Federal  bureaucrats. 

2.  Hendry,  I.  A.,  Editor,  "Reports  of  Clinical  Chemistry  Surveys," 
Board  of  Education  of  the  College  of  Pathologists  of  Australia, 
Sydney  meeting,  August  1969. 

The  program  reported  here  was  based  on  the  premise  that  most  of  the 
testing  done  in  clinical  laboratories  is  on  specimens  from  people  in 
relatively  good  health.    The  test  plan  was  designed  to  provide  information 
(a)  about  the  level  of  laboratory  daily  precision,  and  (h)  on  the  ability 
of  the  laboratories  to  maintain  precision  over  a  three-month  interval. 

Two  classes  of  data  were  collected,  the  one  from  a  conventional  survey 
where  two  sets  of  different  sera  were  mailed  for  testing,  and  the  other 
consisting  of  two  groups  of  data  from  tests  on  patients'  specimens,  each 
group  containing  the  results  of  90  tests.    The  sets  and  groups  consisted 
of  data  taken  in  the  same  two  time  intervals.    The  patients'  test  data 
were  the  same  as  for  the  conventional  survey,  glucose,  urea,  total  protein, 
and  cholesterol.    Unknown  to  the  laboratories,  only  two  different  con- 
centrations were  used,  with  at  least  one  specimen  of  each  in  any  given 
mailing  of  the  six.    Conclusions  cited  are  that: 

(a)  The  specimen  concentrations  in  the  human  sera  remained  constant 
throughout  the  three  months  of  the  study. 

(b)  Information  regarding  the  accuracy  and  precision  of  individual 
laboratories  can  be  obtained  from  patients'  test  data. 

(c)  Surveys  of  laboratories  consisting  of  a  few  survey  unknowns  are 
of  limited  value. 

(d)  Surveys  are  useful  for  determining  the  performance  of  labora- 
tories as  groups,  but  are  of  practically  no  value  for  estimating 
the  performance  of  individual  laboratories. 

(o)  The  information  from  patients'  test  data  is  at  least  as  infor- 
mative as  that  obtained  from  tests  performed  on  survey  samples. 

(f)  Patients'  test  data  is  sufficiently  detailed  to  be  reasonably 
valid  for  estimating  individual  laboratory  performance. 
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This  report  concluded  by  recommending  that  the  College  of  Patholo- 
gists of  Australia  establish  a  computing  service  for  laboratories,  with 
feedback  to  individual  laboratories. 

3.  Hoffman,  Robert  G.,  "Patients'  Test  for  Quality  Control,"  Clinical 
Chemistry,  15:533  (1969). 

A  brief  outline  of  the  method  of  "average-of -normals"  is  given.  It 
consists  in  the  plotting  of  the  mean  +_  the  951  confidence  limits  as  a 
function  of  the  tests  or  time.    Each  point  was  the  average  of  16  tests 
in  one  demonstration,  and  of  10  in  the  other.    In  this  way  it  is  stated 
that  out-of-normal  shifts  can  be  seen. 

4.  Kanon,  Dor,  "Computer  Analyses  Lab  Tests,  Does  Own  Quality  Control," 
Modern  Hospital,  113:105  (1969). 

In  a  form- phase,  all-day  operation,  this  chemistry  laboratory  computer 
updates  patient  files,  standardizes  laboratory  values,  does  quality  con- 
trol, and  produces  cumulative  reports.    The  clinical  chemistry  laboratory 
at  St.  Vincents  Hospital,  New  York,  handles  285,000  tests  a  year.  On- 
line operation  has  form  phases  that  describe  in  detail  the  day's  work. 
The  use  of  standards  is  described,  as  is  the  reliance  on  quality  control 
programs.    Loss  of  sensitivity  from  reagent  deterioration,  and  increase 
in  standard  deviation  from  dialyzer  diaphragm  deterioration  have  been 
detected.    Monthly  cost  is  about  $7,000.    Biochemistry  production 
increased  51%  in  the  first  six  months  of  operation  with  no  increase  in 
personnel . 

5.  Kaufmann,  William  and  six  co-authors,  "Clinical  Laboratory  Performance 
Experience  with  the  New  York  State  Clinical  Laboratory  Program,"  New 
York  State  Journal  of  Medicine,  15:1989  (1969). 

In  1964  the  New  York  State  Legislature  passed  the  Laboratory  Improve- 
ment Act,  which  requires  all  clinical  laboratories  to  operate  under  permit 
and  to  have  in  charge  an  individual  who  holds  a  valid  certificate  of 
qualification.    There  were  430  clinical  laboratories  included  in  the  1967 
to  1968  survey,  made  up  of  229  hospital  laboratories  and  201  independent 
laboratories,  privately  or  municipally  owned.    The  six  areas  of  bacteri- 
ology, mycobacteriology,  mycology,  serology,  hematology,  and  clinical 
chemistry  were  covered. 

6.  McSwiney,    R.  R.  and  D.  A.  Woodrow,  "Types  of  Errors  Within  a 
Clinical  Laboratory,"  Journal  of  Medical  Laboratory  Technology, 
26:340  (1969). 

In  this  study  all  errors  were  counted  over  four  weeks  covering  a 
total  of  approximately  7,000  tests.  , There  were  163  errors  found,  63  of 
them  by  senior  technicians  and  57  by  pathologists.    In  addition,  two  are 
known  to  have  "escaped"  into  the  wards.    "Organizational"  errors  consisted 
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mostly  o£  omitted  or  delayed  analyses,  and  accounted  for  341  of  the  total 
Over  half  the  errors  (54%)  were  clerical  in  origin.    A  normal  day's  work 
produced  15  to  20  errors.    Errors  are  made  more  than  realized;  they  are 
forgotten  as  a  familiar  part  of  life  but  are  not  permissible  within  the 
clinical  laboratory. 

7.  Sideman,  L. ,  D.  T.  Wilson,  and  J.  J.  Murphy,  Jr.,  "A  Survey  of 
Immunohema to logical  Proficiency  Among  131  Clinical  Laboratories  in 
Pennsylvania Health  Laboratory  Science,  6:156-161  (1969). 

Survey  objectives  were:     (1)  to  uncover  laboratory  errors  in  the 
identification  of  major  blood  groups  and  Rh  type,  and  (2)  having  found 
the  errors,  to  offer  consultation  and  training  in  order  to  eliminate 
the  causes.    The  overall  improvement  shown  in  a  subsequent  re-evaluation 
of  these  same  laboratories  emphasizes  the  value  of  a  comprehensive 
evaluation  program  which  includes  interlaboratory  studies,  consultative 
services,  training,  and  re- evaluation. 

8.  Seracci,  R. ,  "Factors  Affecting  Accuracy  and  Precision  in  Clinical 
Chemistry,"  American  Journal  of  Clinical  Pathology,  52:161-166  (1969) 

Accuracy  and  precision  in  four  clinical  chemistry  determinations 
(glucose,  urea,  total  cholesterol,  total  proteins)  have  been  investigated 
in  a  survey  of  404  Italian  laboratories.    Inaccuracy  and  imprecision  were 
found  which  appear,  under  survey  conditions,  to  be  little  related  to 
factors  such  as  class  and  size  of  laboratories,  qualifications  and 
seniority  of  analyst,  and  method  of  analysis.    This  would  imply  that 
acting  upon  these  factors  is  less  important  in  improving  laboratory 
performance  than  is  the  systematic  use  of  quality  control  methods. 

9.  Skendzel,  L.  P.  and  W.  J.  Youden,  "A  Graphic  Display  of  Interlabora- 
tory Test  Results,"  American  Journal  of  Clinical  Pathology,  51:161 
(1969). 

A  simple  graphic  system  worked  out  by  Youden  is  described  and 
advantages  noted.    One  requirement  for  this  system  is  that  laboratories 
analyze  two  samples  for  the  same  constituent.    Reference  laboratory 
results  are  plotted  against  each  other,  with  intersecting  lines  passing 
through  each  mean,  the  SD  is  multiplied  by  3.035  to  obtain  the  radius 
of  a  circle  that  is  drawn  with  the  intersection  of  the  means  as  the 
center.    This  circle  will  then  include  991  of  the  values  of  reports  for 
the  two  samples.    Two  tangents  with  positive  slope  at  45°  are  drawn  to 
the  circle  one  at  each  end  of  the  diameter.  .  Pairs  of  values  from  other 
participants  are  then  used  to  plot  additional  points. 
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1.  Barnett,  Roy  N.  and  W.  J.  Youden,  "A  Revised  ScJieme  for  the  Compari- 
son of  Quantitative  Methods ," American  Journal  of  Clinical  Pathology, 
54:454-462  (1970). 

The  authors,  on  the  basis  of  experience  gained  in  evaluation  of  50 
different  "kits"  for  chemical  analysis,  have  developed  a  revised  and 
simplified  method  for  conducting  such  evaluations.    The  scheme,  which  is 
applicable  for  any  comparison  of  quantitative  methods ,  describes  repro- 
ducibility studies  (20  once-a-day  analyses),  recovery  studies  in 
triplicate  at  two  or  three  levels,  and  comparison  studies  with  patient 
serum,  all  performed  by  the  test  method  and  a  reference  method.  Statis- 
tical calculations  and  criteria  of  acceptability  are  described  in  detail 
and  a  complete  example  given. 

2.  Barnett,  Roy  N. ,  W.  Harold  Civin,  and  Irwin  Schoen,  "Multiphasic 
Screening  by  Laboratory  Tests  -  An  Overview  of  the  Problem,"  American 
Journal  of  Clinical  Pathology,  54:483-492  (1970). 

The  authors  considered  the  problems  of  jnultiphasic  screening  by 
laboratory  tests  in  the  framework  of  the  concept  "Total  Quality  Control 
in  the  Clinical  Laboratory,"    After  reviewing  the  data,  the  problems,  and 
the  opinions  of  others, they  drew  the  following  conclusions.    It  is 
entirely  premature  to  recommend  the  routine  use  of  extensive  multiple 
screening  tests  for  either  hospital  admission  or  general  populations, 
considering  the  present  ignorance  of  physicians  about  every  one  of  the 
six  categories  we  used.    At  present  these  multiple  screening  tests  should 
be  considered  research  rather  than  service  activities.    Although  the 
laboratory  problems  in  testing  are  still  formidable,  they  are  being 
solved  far  more  rapidly  than  are  the  problems  relating  to  the  medical 
usefulness  of  the  test  results.    There  is  an  urgent  need  for  approximately 
controlled,  large  scale,  multidisciplinary  studies  to  answer  the  basic 
questions  concerning  the  utility  of  the  data.    We  cannot  accept  the 
assumption  that  the  production  of  huge  volumes  of  "screening"  information 
will  by  itself  contribute  to  human  knowledge  or  health.    A  selected 
bibliography  is  appended. 

3.  Eilers,  Russell  J.,  "Total  Quality  Control  for  the  Medical  Labora- 
tory:   The  Role  of  the  College  of  American  Pathologists  Survey 
Program,"  American  Journal  of  Clinical  Pathology,  54:435  (1970). 

Six  phases  of  Quality  Control  are  set  forth  as  follows: 

I.  Selection  of  proper  test  method 

II.  Selection  of  proper  standards  control  and  SRM's 

III.  Internal  Quality  Control  of  process  and  proficiency  testing 

IV.  Proper  format  of  results 

V.  Reliability  and  interpretation  of  results 

VI.  Inspection  and  accreditation. 
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4.  Evenson,  M.  A.,  "The  Need  for  Analytical  Chemistry  in  Clinical 
Chemistry  Training  Programs,"  Analytical  Chemistry,  42:53A  (1970). 

The  shortage  o£  adequately  trained  clinical  chemists  will  soon  become 
a  manpower  crisis,  but  recent  Federal  fund  tightening  activities  have 
dampened  enthusiasm  for  new  programs.    There  are  1,600  persons  in  the 
U.S.  who  have  more  than  a  bachelor's  degree  in  chemistry  and  are  members 
of  the  American  Association  of  Clinical  Chemists.    The  20  most  common 
clinical  chemistry  tests  are  listed.    Each  clinical  chemistry  laboratory 
doubles  its  own  size  every  five  years.    A  short  history  of  clinical  is 
given,  starting  with  Paracelsus  in  1526.    Comments  on  the  administrative 
and  academic  arrangements  usually  formed  by  clinical  chemists  are  made. 
Need  for  analytical  chemical  excellence  in  clinical  chemistry  laboratories 
is  stressed. 

5.  Gavan,  Thomas  L.  and  John  W.  King,  "An  Evaluation  of  the  Microbiology 
Portions  of  the  1969  Basic,  Comprehensive,  and  Special  College  of 
American  Pathologists  Proficiency  Testing  Surveys,"  American  Journal 
of  Clinical  Pathologists,  54:514  (1970). 

The  proficiency  testing  program  of  the  CAP  was  markedly  expanded  in 
1969  by  adding  three  new  surveys:    Special  Bacteriology,  Special  Myco- 
bacteriology,  and  Special  Mycology.    The  use  of  mycobacterial  and 
mycologic  specimens  in  the  Comprehensive  and  Basic  Surveys  was  expanded. 

The  proficiency  testing  surveys  of  the  CAP  and  those  of  other  agencies 
are  limited  to  evaluating  a  laboratory's  ability  to  identify  a  given 
organism  or  organisms.    No  satisfactory  methods  exist  for  large  scale 
surveys  in  microbiology,  based  on  simulated  clinical  specimens,  to  enable 
laboratory  proficiency  evaluation  in  isolation  techniques  as  well  as  the 
identification  of  bacteria  and  fungi. 

In  all,  83  specimens  of  various  bacteria  and  fungi  were  distributed, 
with  19  being  common  to  two  or  more  surveys.    A  total  of  34,908  reports 
from  participants  have  been  evaluated.    Methods  of  rating  participant 
performance  in  terms  of  referee  performance  are  given,  and  the  results  of 
both  are  presented  in  seven  tables.    In  general,  80%  or  more  of  the  par- 
ticipant laboratories  were  able  to  identify  the  pyogenic  bacteria  submitted 
in  a  good  or  acceptable  manner.    The  exceptions  are  cited  and  discussed. 
Mycobacteria  were  acceptably  identified  by  significantly  fewer  labora- 
tories than  the  pyogenic  bacteria.    Fungi  were  more  poorly  identified 
than  pyogenic  bacteria,  but  were  correctly  identified  more  frequently 
than  the  mycobacteria. 

The  quality  and  scope  of  survey  material  have  been  improved,  as  well 
as  the  results  reported  by  participants.    Improved  techniques  for  the 
conduct  of  such  surveys  are  given,  including  specimen  checking  and  in 
freeze  drying  and  handling  specimens. 
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6.  Gilbert,  Roy  K. ,  "Analysis  of  Results  of  the  1969  Comprehensive 
Chemistry  Survey  of  the  College  of  American  Pathologists,"  American 
Journal  of  Clinical  Pathology,  54:463-482  (1970). 

The  samples  of  this  survey  consisted  of  eight  vials  mailed  out  in 
pairs  on  four  occasions  during  the  year.    The  results  are  described  for 
analyses  for  glucose,  calcium,  total  bilirubin,  potassium,  sodium,  urea 
nitrogen,  uric  acid,  creatinine,  chloride,  phosphorous,  protein-bound 
iodine,  and  total  protein.    The  total  number  of  analyses  was  140,730. 
The  results  were  categorized  by  method  used,  and  a  clear  pattern  of 
relationship  was  apparent  between  the  method  and  both  the  mean  values 
and  the  standard  deviations.    Some  methods  showed  a  constant  relation- 
ship in  mean  values  with  one  method  higher  or  lower  throughout  the  series. 
In  others  there  was  variability  and  the  relationship  changed.  This 
fluctuation  could  occasionally  be  traced  to  interaction  with  other  con- 
stituents.   The  methods  also  showed  a  relationship  to  the  standard 
deviations  throughout  the  series,  with  one  method  usually  higher  or  lower 
than  another.    The  relationship  tended  to  be  more  constant  than  that  for 
the  mean  values. 

Variance  is  ascribed  to  three  sources :    the  basic  analytical  per- 
formance, calibration  standards,  and  systematic  differences  between 
laboratories;  these  are  discussed  at  length.    The  results  show  that  many 
methods  give  mean  values  that  differ  from  those  of  other  methods.  The 
data  also  show  that  this  is  not  a  constant  relationship  and  that  inter- 
action between  constituents  occurs.    The  spread  of  values  is  wider  by 
some  methods  than  others,  and  the  automated  method  tends  to  produce  a 
narrower  range  although  a  survey  offers  some  bias  in  favor  of  automated 
means. 

7.  Hanson,  Daniel  J. ,  "Suggested  Definitions  for  Clinical  Laboratory 
Standards  and  Reference  Materials,"  American  Journal  of  Clinical 
Pathology",  54:451  (1970). 

The  CAP  is  concerned  about  the  existing  confusion  regarding  definition 
and  nomenclature  of  standards  that  are  applicable  to  clinical  pathology. 
The  situation  is  an  impediment  to  laboratory  performance  evaluation  and 
to  the  application  of  standards  ^d  controls  in  the  clinical  laboratory. 

There  is  an  impelling  and  immediate  need  for  agreement  on  terms  and 
definitions  by  pathologists,  technologists,  and  manufacturers  to  resolve 
confusion  and  permit  evaluation  of  reports.    Definitions  are  presented 
and  discussed  for  Primary  Standard,  Primary  Volumetric  Standard,  and 
Secondary  Volumetric  Standard.    The  clinical  versions  of  these  standards 
are  also  presented,  as  is  the  term  Reference  Material  or  Reference 
Control,  originally  referred  to  by  Radin  as  "reference  samples." 
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8.  Koepke,  John  A. ,  " Iiranunohematology  Proficiency  Testing,  1966-1969," 
American  Journal  of  Clinical  Pathology,  54:508-511  (1970). 

Immunohematology  proficiency  testing  as  measured  by  the  Laboratory 
Surveys  of  the  College  of  American  Pathologists,  from  1966  to  1969,  is 
reviewed.    During  this  period  there  has  been  a  gradual  expansion  of  this 
portion  of  the  survey.    Over  the  years  studied  there  has  been  an  evident 
improvement  in  ABO  grouping  and  the  detection  of  atypical  antibodies. 
Crossmatching  has  continued  to  be  well  done;  however,  performance  of  Rh 
typing  has  been  spotty,  presumably  due  to  variation  in  typing  sera  or 
red  cell  testing  specimens,  or  both. 

9.  Reed,  A.  H. ,  "Use  of  Patient  Data  for  Quality  Control  of  Clinical 
Laboratory  Tests,"  Clinical  Chemistry,  16:129  (1970). 

The  average-of -normals  quality- control  method  is  examined  mathemati- 
cally.   Two  modifications  are  proposed.    After  modification,  the  essential 
unchanged  feature  of  the  method  is  the  idea  of  censoring  the  laboratory 
test  results  so  that  only  means  of  test  values  inside  the  normal  range  are 
plotted  on  the  quality- control  chart.    Several  limitations  of  the 
modified  method  are  discussed,  and  it  is  concluded  that,  although 
censoring  does  improve  the  probability  of  detecting  drift,  more  statis- 
tical research  and  further  modifications  are  needed  before  the  average- 
of-normals  method  will  be  useful  for  quality  control  in  the  clinical 
laboratory. 

10.  Rodwell,  V.  W. ,  Editor,  "Test  Theory,  Proficiency  Tests,  and 
Unevaluated  Tests,"  Test  of  the  Month  ISb.  1 ,  Proficiency  Testing 
Service,  American  Association  of  Bioanalysts  (1970). 

Research  is  needed  to  gain  a  greater  understanding  of  procedure  and 
objectives  in  carrying  out  proficiency  testing  and  on-site  inspections. 
PT  under  Medicare  is  cited  as  is  CLIA  '67.    Cost  problems  of  private 
groups  are  noted.    CAP  participant  cost  is  $881/year,  while  CDC  will  do 
the  same  thing  for  less  than  $125. 

There  is  little  understanding  of  the  theory  of  testing.    It  was 
first  given  formal  expression  by  Dr.  Reuben  L.  Kahn:    first,  a  procedure 
is  employed  to  derive  a  probability  statement  with  regard  to  any  deter- 
mination; second,  group  validity  in  regard  to  any  determination  is  always 
more  reliable  than  individual  validity;  third,  the  parameters  involved  in 
any  test  are  sensitivity,  specificity,  and  reproducibility  of  results. 
An  exception  is  blood  typing  where  there  is  no  room  for  error  in  deter- 
mining ABO  and  Rh  types. 
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11.  Schaeffer,  M. ,  D.  Widelock,  P.  S.  May,  S.  Blatt,  and  M.  E.  Wilson, 
"Tlie  Clinical  Laboratory  Improvement  Program  in  New  York  City:  II. 
Progress  After  Five  Years  of  Experience,"  Health  Laboratory  Science, 
7:242  (1970). 

In  previous  studies,  clinical  laboratories  in  New  York  City  were 
analyzed  with  respect  to  organization  and  technical  competence.  From 
that  evaluation  it  became  apparent  that  laboratories  performing  satis- 
factorily are  those  that  employ  full-time  competent  supervisors.  This 
experience  led  to  efforts  to  in^rove  the  performance  of  consistently  poor 
laboratories. 

Two  methods  designed  to  improve  the  proficiency  of  clinical  labora- 
tories in  either  bacteriology  or  chemistry  were  evaluated.  On-site 
consultation  by  experts  in  these  two  disciplines,  and  workshop  instruction 
were  studied  for  their  effect  on  results  of  proficiency  testing  of  labora- 
tories selected  for  assistance.    The  latter  were  compared  with  similar 
laboratories  treated  routinely,  without  advice  or  instruction. 

Analysis  of  results  appears  to  indicate  that  some  improvement  in 
performance  was  achieved  simply  through  repeated  proficiency  tests, 
laboratories  participating  in  bacteriology  workshops  showed  improved 
performance  as  compared  with  control  laboratories.    However,  laboratories 
enrolled  in  clinical  chemistry  workshops  showed  no  greater  improvement 
than  control  laboratories. 

12.  Schoen,  I.,  "Guidelines  to  Laboratory  Management ," California  State 
Department  of  Public  Health,  Some  Proficiency  Testing  Experiences 
in  California  for  1968,  Volume  D-1  (1970) . 

Proficiency  testing  is  mandatory  for  many  licensed  laboratories  in 
California.    There  were  226  laboratories  in  the  State  Comprehensive 
Survey,  and  201  in  the  Basic  Survey;  the  Basic  Survey  includes  sufficient 
specimens  and  examinations  to  meet  the  mandatory  PT  requirements,  while 
the  comprehensive  survey  includes  more  specimens  and  examinations  than 
required  by  the  mandatory  regulations.    Overall,  the  performance  of 
California  laboratories  and  all  participating  laboratories  was  very 
similar  with  no  notable  difference  in  the  data.    The  means  of  reporting 
and  selection  of  specimens  by  current  survey  techniques  appears  to  lead 
to  unsatisfactory  erroneous  mixed-up  reports  involving  up  to  1.5%  of 
reports,  and  up  to  51  of  laboratories. 

Automation  seems  more  precise  for  some  procedures.    This  precision 
might  be  due  not  to  automation  itself,  but  may  be  more  appropriately  due 
to  standardizing  or  controlling  instrumentation  methods,  reagents,  and 
standards.    A  "correct"  report  on  a  survey  specimen  does  not  mean  a 
"correct"  interpretation  of  a  report  on  a  patient's  specimen  from  a  labora- 
tory with  an  incorrect  set  of  normal  values.    Medical  Usefulness  Criteria 
as  well  as  data  from  Peers  or  Reference  Laboratories  should  be  utilized 
in  evaluating  performance. 
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1970  (continued) 


Local  or  regional  data  accumulation  and  evaluation  should  be  consid- 
ered to  achieve  more  prompt  evaluation  with  follow-up  testing  and  con- 
sultation for  unsatisfactory  performance. 

To  best  serve  the  interest  of  reliable,  better,  and  improved  labora- 
tory services  by  a  specific  laboratory  for  the  patient,  a  program  of 
standards  for  quality  control  analogous  to  standards  for  a  Blood  Trans- 
fusion Service  should  be  considered. 

13.  Skendzel,  Laurence  P.,  "Guidelines  for  the  Design  of  Laboratory 

Surveys,"  American  Journal  of  Clinical  Pathology,  54:437-447  (1970). 

This  report  outlines  the  system  used  in  the  College  of  American 
Pathologists  National  Survey  Program  for  medical  laboratories  with  special 
emphasis  on  data  handling  and  evaluation  of  a  laboratory's  performance. 
The  statistical  techniques  for  data  analysis  of  quantitative  procedures 
include:     (1)  exclusion  of  widely  deviant  values,  (2)  calculation  of  mean, 
SD,  and  CV  after  exclusion  of  deviant  values  to  describe  the  distribution 
of  values,  (3)  calculation  of  the  average  CV  to  summarize  the  distribution 
of  values  from  several  surveys,  (4)  reliance  on  the  two-sample  plot  and 
the  "t"  test  to  combine  results  derived  by  different  methods.  Evaluation 
of  laboratory  performance  is  based  on  the  "State-of-the-Art"  concept  which 
uses  participant's  values  to  set  acceptance  and  not  acceptable  limits. 
Other  systems  for  evaluation  are  discussed. 
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Appendix  C.    Temporal  Effect  of  Continued  Participation 
in  the  CDC  Proficiency  Testing  Program 
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Table  C3  lists  the  most  frequently  used  analytical  methods  for  each 
of  the  constituents  in  the  CDC  proficiency  testing  data  base. 


Constituent 
Glucose 
Creatinine 

Urea  Nitrogen 
Uric  Acid 

Total  Protein 

Cholesterol 

Triglyceride 
Calcium 

Sodium 
Potassium 

Chloride 


Table  C3.    Most  Popular  Analytical  Methods 

Methods 
(1)  AutoAnalyzer;  (2)  SMA 

(1)  Alkaline  picrate  (w/o  Lloyds); 

(2)  AutoAnalyzer 

(1)  AutoAnalyzer;  (2)  SMA 

(1)  AutoAnalyzer;  (2)  SMA;  (3)  Phosphotungstate- 
carbonate 

(1)  AutoAnalyzer;  (2)  SMA;  (3)  Biuret,  serum 
standard;  (4)  Refractive  index 

(1)  AutoAnalyzer;  (2)  SMA;  (3)  Direct  serum 
(acetic  anhydride) 

(1)  Acetylacetone  reagent;  (2)  Enzymatic 

(1)  AutoAnalyzer;  (2)  SMA;  (3)  Atomic  . 
Absorption 

(1)  Flame  Photometry;  (2)  SMA 

(1)  Flame  Photometry;  (2)  AutoAnalyzer; 

(3)  SMA 

(1)  Cotlove  Titrator;  (2)  AutoAnalyzer; 

(3)  SMA 


Hematocrit 

Red  Blood  Cell  Count 
Hemoglobin 


(1)  Micro- Hematocrit 
(1)  Coulter  counter 

(1)  Coulter  counter,  model  S;  (2)  Cyanmethemo- 
globin  (manual) 
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Appendix  D.    Measuring  Laboratory  Performance  in 
the  CDC  Proficiency  Testing  Program 

In  very  general  terms,  clinical  laboratory  analyses  fall  into  two 
procedural  categories;  quantitative  tests  (chemistry,  hematology,  differ- 
ential blood  films,  some  seriology  and  the  radioisotopes)  and  qualitative 
tests  (bacteriology,  cytology,  urinalysis,  some  serology,  and  most  of 
immunohematology,  and  toxicology).    The  next  few  paragraphs  discuss  the 
CDC  proficiency  testing  program  in  these  two  basic  categories. 


Quantitative  Tests 

Reference  laboratories,  usually  100  or  more,  are  used  as  prescribed 
by  regulation  (Federal  Register  No.  253,  Vol.  33,  December  31,  1968, 
p.  20047).    Criteria  for  satisfactory  performance  for  each  sample  are 
enumerated  in  paragraph  74.42;  they  basically  require  the  use  of  three 
subsets  of  limits.    The  first  subset  encompasses  the  central  95  percentile 
calculated  from  the  results  of  all  applicant  and  licensed  laboratories. 
Laboratory  results  which  are  "obviously  deviant  from  the  other  results" 
are  not  used.    No  specific  criteria  or  procedures  are  provided  for  deter- 
mining which  values  are  "obviously  deviant,"  however,  there  is  a  stipu- 
lation that,  for  any  sample,  no  more  than  five  percent  of  the  results 
are  to  be  discarded  on  the  basis  of  being  "obviously  deviant."  The 
second  subset  of  limits  is  determined  from  the  reference  laboratory  results; 
they  encompass  all  the  results  and  simply  consist  of  the  highest  and 
lowest  values  (the  same  procedure  is  followed  for  outliers  as  was  just 
described) .    The  final  subset  of  limits  (for  clinical  requirements)  is 
centered  on  the  median  reference  laboratory  result.    "They  encompass  one- 
half  of  the  normal  population  range,  when  the  sample  is  in  the  normal 
range,  or  one-half  of  the  appropriate  range  in  the  case  of  a  sample  which 
is  outside  the  normal  range."    The  normal  population  range  or  other 
appropriate  range  is  determined  by  the  Secretary,  DHEW.    The  results  for 
applicant  and  licensee  laboratories  must  fall  between  the  lowest  lower 
limit  and  highest  upper  limit  of  the  three  subsets  of  limits  just 
described. 

For  chemistry  and  hematology,  a  score  of  three  is  assigned  to  all 
laboratory  results  within  the  tightest  combination  of  limits  around  the 
target  value.    A  score  of  two  is  assigned  to  all  laboratory  results 
within  the  next  tightest  combination  of  limits  (but  not  within  the 
tightest  combination) ,  and  a  score  of  one  is  assigned  to  all  laboratory 
i^esults  falling  within  the  outermost  limits  determined  from  the  three 
subsets  of  limits  (but  not  within  the  two  previously  described  limits) . 
All  results  falling  outside  of  the  outermost  limits  are  assigned  a  score 
of  -1.    All  laboratory  results  having  a  score  of  one  or  greater  are  con- 
sidered acceptable. 

The  scoring  methods  for  differential  blood  films  and  non- syphilis 
serology  are  slightly  more  complicated.  They  can  be  found  in  a  report 
entitled  "Development  of  Grading  Methods,"  a  report  of  the  Study  Group 
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for  the  Development  of  Grading  Methods,  presented  at  the  First  National 
Conference  on  Proficiency  Testing,  Atlanta,  Georgia,  October  4-6,  1971. 


Qualitative  Tests 

Qualitative  tests  differ  from  the  quantitative  tests  in  that  in 
addition  to  using  a  select  group  of  three  or  more  reference  laboratories, 
the  samples  are  also  sent  to  ten  or  more  referee  laboratories.  The 
reference  laboratories  analyze  the  samples  in  great  detail  while  the 
referee  laboratories  handle  the  samples  in  the  same  manner  as  do  the 
licensed  laboratories  being  tested. 

The  grading  procedures  for  the  qualitative  tests  are  much  more 
complex  than  those  which  were  described  for  the  quantitative  tests.  The 
reader  is  directed  to  pages  20047-8,  in  the  December  31,  1968  Federal 
Register  for  a  complete  discussion  of  this  subject. 
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Appendix  li.    Mean  Measures  of  Relative  Performance 

The  tables  included  in  this  appendix  list  mean  measures  of  relative 
performance  (MRP)  which  have  been  found  to  be  significantly  different  from 
each  other  or  from  other  subgroups.    A  separate  table  appears  for  each  of 
the  14  constituents.    In  each  instance j  the  difference  between  individual 
means  (calculated  from  all  samples  covered  in  the  two-year  period)  or 
subgroups  was  significant  at  the  0.001  level  of  probability.    This  signif- 
icance was  based  on  the  approximate  F-test  made  necessary  by  significantly 
heterogeneous  variances  (p  <  .001). 


Table  El.  Glucose 

At  least  15  laboratories  used  each  of  the  following  six  analytical 
methods.    Means  which  are  underscored  by  the  same  line  are  not  signif- 
icantly different  from  each  other.    Hence,  methods  numbered  7,  9,  4,  8 
and  6  are  in  the  same  subgroup,  as  are  methods  6  and  3;  while  method  3 
is  significantly  different  from  methods  7,  9,  4  and  8. 


Analytical  Method 

Mean  Relative 
Performance 

Number  of 
Laboratories 

#7 

-  AutoAnalyzer 

-1.51 

86 

#9 

-  Other  Automated 

-1.31 

20 

#4 

-  Direct  Serum 

-  .2% 

47 

#8 

-  3VIA  or  Neocuproine 

0.2% 

93 

#6 

-  Other  Manual 

1.7% 

21 

#3 

-  Ortho  Toluidine 

5.5% 

17 

7,  9,  4,  8,  6,  3 


Table  E2.  Creatinine 

There  were  five  methods  which  were  used  by  more  than  15  laboratories 
each.  These  five  methods  fall  into  four  subgroups  with  only  one  instance 
of  commonality. 

Analytical  Method  Mean  Relative  Number  of 

Performance  Laboratories 


#6  -  Other  Manual  -22.3%  17 
#2  -  Alkaline  Picrate, 

w/o  Lloyds  -  5.1%  149 

#8  -  SMA  -  3.5%  22 

#7  -  AutoAnalyzer  -  3.3%  53 
#1  -  Alkaline  Picrate, 

Lloyds  16.1%  29 

6    2    8,  7  1 


50 


Table  E3.    Urea  Nitrogen 


Analytical  Method  Mean  Relative  Number  o£ 

Performance  Laboratories 


#8  -  SMA  -1.91  97 

#1  -  Urease-Nesolerization  -  .7%  15 

#7  -  AutoAnalyzer  -  .6%  80 

#2  -  Urease- Berthelot  2.8%  23 

#3  -  Diacetyl  Reaction  3.6%  35 

#6  -  Other  Manual  7.2%  15 

#9  -  Other  Automated  39.0%  16 

#5  -  Kits  39.7%  17 


8,  1,  7    2,  3    6    9,  S 


Table  E4.    Uric  Acid 

Analytical  Method  Mean  Relative  Number  of 

Performance  Observations 


#6  -  Other  Manual  -8.6%  19 
#1  -  Phosphotungstate- 

cyonide  -3.8%  18 
#2  -  Phosphotungstate- 

carbonate  -3.0%  73 

#8  -  SMA  -1.6%  94 

#7  -  AutoAnalyzer  0.7%  64 


6    1,  2    8  7 


Table  E5.  Calcium 


'  Analytical  Method  Mean  Relative  Number  of 

Performance  Observations 


#7  -  AutoAnalyzer  -1.0%  32 

#5  -  Flame  Photometer  -  .9%  24 

u  -  mk  -  .1%  112 

#2  -  EDTA  Titrimetric  0.4%  48 

#3  -  EDTA  Colorijnetric  1.2%  27 

#4  -  Atomic  Absorption                .2.0%  27 

#6  -  Other  Manual  2.4%  48 

#9  -  Other  Automated  2.6%  20 


7,  5    8    2    3    4,  6,  9 
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Table  E6.  Chloride 


Analytical  Method 

Mean  Relative 

Number  of 

Performance 

Observations 

//8 

-  SMA 

-.51 

20 

-  Cotlove  Titrator 

-  4% 

54 

#3 

-  Other  Titrimetric 

-A% 

44 

#1 

-  Schales 

0.6% 

111 

#7 

-  AutoAnalyzer 

0.7% 

33 

IT  J 

-  Kits 

1.0% 

76 

8,  2    3    1,  7  5 

'                         Table  E7.    Total  Protein 

Analytical  Method 

Mean  Relative 

Number  of 

Performance 

Observations 

#7 

—    Ai  Tt'oA'n^i  1  vTi^T* 

-  7?; 

32 

#2 

-  Biuret,  Serum 

-.6% 

79 

#8 

-  SMA 

-.5% 

114  ■ 

TT  O 

-  Refractive  Index 

0.6% 

7     ?     8  =1 

Table  E8.  Potassium 

Analytical  Method 

Mean  Relative 

Number  of 

Performance 

Observations 

#9 

-  Other  Automated 

-4.8% 

16 

#7 

-  AutoAnalyzer 

-  .3% 

16 

#1 

-  Flame  Photometry 

0.4% 

169 

#8 

-  3^ 

0.9% 

17 

9    7,  1,  8 

Table  E9.  Sodium 

Analytical  Method 

Mean  Relative 

Number  of 

Performance 

Observations 

#9 

-  Other  Automated 

-.7% 

15 

#1 

-  Flame  Photometry 

0.3% 

168 

#8 

-  SMA. 

0.5% 

18 

#7 

-  AutoAnalyzer 

0.6% 

18 

9    1    8,  7 
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Table  ElO.  Cholesterol 


Analytical  Method 


#7  -  AutoAnalyzer  Extract 
#3  -  Ferric- Sulfuric 
#8  -  SMA 

#5  -  Direct  Serum, 

Ferric- Sulfuric 

#4  -  Direct  Serum, 

Acetic  Anhydride 

#6  -  Other  Manual 

#9  -  Other  Automated 


Mean  Relative 
Performance 


-3.3 
-2.5 
-2.5 


7  3 


■2A\ 
■2.V 
0.6! 


5  4 


6,  9 


Number  of 
Observations 

67 
18 
97 

21 

87 
48 
18 


Analytical  Method 


Table  Ell.  Triglycerides 

Mean  Relative 
Performance 


#1  -  Ace tyloce tone  Reagent 
#2  -  Chromotropic  Acid 
#5  -  Other 


1.21 
10.0% 
13.7% 


Number  of 
Observations 

132 

18 
28 


12  5 


Table  E12.  Hemoglobin 

Analytical  Method  Mean  Relative  Number  of 

Performance  Observations 


#9  -  Other  Automated  -2.11  47 

#1  -  Cyanmethemoglobin  -1.9%  124 

#6  -  Coulter  S  2.1%  75 

#8  -  SMA  2.3%  20 

#5  -  Other  Manual  21.1%  16 

9,  1    6,  8  5 
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Table  El 3.    Red  Blood  Cell  Count 


Analytical  Method  Mean  Relative  Number  of 
  Performance  Observations 

#3  -  SMA 
#2  -  Coulter 
#5  -  Other  Automated 
#1  -  Manual 

3    2    5  1 


-.6% 

0.7% 
2.6% 
4.9% 


16 
126 
15 
30 


Table  E14.  Hematocrit 

Analytical  Method  Mean  Relative  Number  of 

Performance  Observations 


#3  -  Other  -4.8%  15 

#2  -  Micro  Hematocrit     ^  0.4%  159 

■     ^'3  2 
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